Daucloud's Blog

减脂饮食笔记

Mon, 07 Sep 2026 00:00:00 +0800

整理自 B 站视频《健身新手的减肥减脂完全手册》。原来是几张图片笔记，翻起来不方便，重新排成文字放在这里。

下文的 g/kg 都指“每公斤体重每天吃多少克”。这些数字是起点，不是处方，按后面的规则两周一调就行。

TL;DR

按体重和训练情况定蛋白质、脂肪、碳水的起始量。
两周看一次体重。在掉就不动；不掉就只减碳水，每次 0.3–0.5 g/kg，蛋白质和脂肪不碰。
训练日碳水集中在训练前后，其他餐少吃或不吃碳水。
休息日碳水降到 1.5–2 g/kg，每餐均分。
有氧等饮食稳定了再加，前期一周最多一次。

起始配额

力量训练日

人群	碳水	蛋白质	脂肪
男性	3–3.5 g/kg	新手 1.5，有基础 2 g/kg	0.8 g/kg
女性	2.5–3 g/kg	1.5 g/kg	0.8 g/kg
不健身	2–2.5 g/kg	1.2 g/kg	0.8 g/kg

以 90 kg 男性为例：蛋白质 135 g（新手）或 180 g（有基础），脂肪 72 g，碳水 270–315 g。

大体重

BMI 偏高的人不要直接套上面的表，起点要低一档。单位都是 g/kg：

BMI	男性碳水	女性碳水	健身者蛋白质	不健身者蛋白质	脂肪
>28	2.5	2.1	1.2	1.0	0.6
>32	2.0	1.7	1.0	0.8	0.5

休息日或只做有氧

碳水 1.5–2 g/kg，蛋白质和脂肪照训练日。90 kg 对应 135–180 g。

两周一调，只动碳水

两周掉 1.5%–2.5% 体重算正常，不用改。90 kg 对应 1.35–2.25 kg。
头两周掉得快多半是水和糖原，别当成掉脂肪的速度。
两周没掉，碳水减 0.3–0.5 g/kg，再看两周。蛋白质和脂肪不动。90 kg 一次减 27–45 g。
还在掉就别减了。
视频给的下限：非比赛不要低于 2 g/kg，大体重除外。90 kg 就是 180 g。

训练日怎么吃

下面的百分比都是全天碳水的比例，不是热量比例。蛋白质和脂肪只在训练后餐有硬要求，其他餐凑够全天总量就行。

以全天 270 g 碳水算：早餐 81 g，训前 54 g，训后 135 g，其他餐基本不剩。

早餐

碳水占全天 30%，不用管 GI。蛋白质 30 g 左右，脂肪 20 g 左右。

食堂版是包子或面条加两个鸡蛋一杯牛奶，方便但不太顶饿。自制版用燕麦、玉米粉、藕粉这类冲蛋白粉，饱腹感好得多。

训练前餐

碳水占 20%，选中高 GI 的，蛋白质随意，脂肪不吃。训练前 15–30 分钟吃，到了健身房才想起来也比空腹练强。面包、香蕉、葡萄干都行，脉动、宝矿力这类含糖饮料也行。

训练后餐

碳水占 50%，高 GI，蛋白质 30–50 g，脂肪 20 g 以内。训练结束半小时内吃。这一餐是要把胰岛素拉上去，尽量少掉肌肉，所以不用怕高 GI。

米饭馒头加瘦肉直接吃正餐最好。来不及就先吃点快碳加蛋白粉垫一下，回头再吃正餐。

其他餐

蛋白质加脂肪加蔬菜，碳水看全天配额还剩多少，剩得少就不吃。瘦肉炒菜配大量蔬菜，主食少放或不放。

按训练时间套用

四种常见训练时段的碳水分配，视频里是四张图，合成一张表：

训练时间	早餐	午饭	晚饭	训练后加餐
早饭后	20%（训前餐）	20%	20%	40%
午饭前	30%	训前 20% + 训后 50%	少吃或不吃	无
晚饭前	30%	少吃或不吃	训前 20% + 训后 50%	无
晚饭后	20%	20%	20%（训前餐）	40%

蛋白质和脂肪的规则不变：训前餐不吃脂肪，训后餐蛋白质 30–50 g、脂肪 20 g 以内，其他餐不严格。零食加餐都可以有，算进全天总量就行。

休息日

每餐都有碳水、蛋白质、脂肪，大致均分，直观做法是每餐一碗饭。碳水总量比训练日少，1.5–2 g/kg。没有训练就不用刻意吃高 GI 去刺激胰岛素。

饿的话先加蔬菜和汤，别把训练日的碳水量搬过来。

有氧

视频对有氧的态度很克制：先把饮食调稳，再考虑加有氧。

	建议
频率	前中期 0–1 次/周，后期 2–3 次/周
时长	每次 40–60 分钟，放在休息日
形式	跑步机、椭圆机、游泳、爬楼、球类都行
心率	120–150，看手表或数 20 秒脉搏
别做	力量训练后接长有氧；每周有氧总量超过 5 小时

空腹有氧是否更有效没有定论。要做的话先吃点蛋白质或 BCAA 垫一下，结束再吃饭。

关于 ALF 中 Bias 引起的 MoE 路由翻转的简要讨论

Sat, 22 Aug 2026 00:42:10 +0800

问题

在 Auxiliary-Loss-Free（ALF）方法中，采用 $\operatorname{TopK}_{s+b}$ 来选择 expert。

correction bias $\mathbf{b}$ 的学习可以改善负载均衡。但是在后续的 gated output 中，仍然采用原始 score $s_i$ 来加权。这就会自然引入一个问题：是否会因为某个 expert 相对于其他 expert 的 bias 差 $b_i-b_j$ 很大，使一个原本 $s_i$ 很小的 expert i 被激活？换言之，明明模型计算出的 $s_i$ 认为 expert i 不应该被选入，它是否仍会由于负载均衡的要求而被选入？

TL;DR

只要 $\mathbf{b}$ 在 expert 之间不完全相同，Top-K 就可能相对于 raw-score Top-K 发生翻转；是否频繁取决于 score gap 与 bias 差的联合分布。仅凭 ALF 机制本身，无法保证翻转概率的上界，也无法保证最大翻转程度 $b_i-b_j$ 的上界。
在 Moonlight-16B-A3B 的真实文本激活样本中，普通翻转很常见：56.24% 的 token-layer 改变了 Top-K 集合；但接近零贡献（ $s_i\approx 0$ ）的事件很少，最后的加权输出中，占比小于 0.01 和 0.001 的 expert 比例分别为 0.1926% 和 0.00063%。
这种换入的影响还有待评估，可能有危害，也可能完全没有危害。直觉上，低权重换入的 expert 自身贡献很小，主要对应潜在计算浪费；但它可能同时排除一个重要的高 raw-score expert，因此整次翻转对模型输出的影响不一定小。当前小规模实验没有显示严重的总体性能问题，但还不足以给出普适的质量结论。

对开源 checkpoint 的实际诊断

bias range 与翻转的确定性关系

先考虑普通 global Top-K。raw-score Top-K 集合记为 $R=\operatorname{TopK}_s$ ，实际集合记为 $A=\operatorname{TopK}_{s+b}$ 。假设 $e_{\mathrm{in}}$ 是被 bias 换入的 expert，即 $e_{\mathrm{in}}$ 属于 $A$ 但不属于 $R$ ；对应地，存在一个被换出的 expert $e_{\mathrm{out}}$ ，即 $e_{\mathrm{out}}$ 属于 $R$ 但不属于 $A$ 。由 Top-K 选择规则可得

s_{\mathrm{in}}+b_{\mathrm{in}}\ge s_{\mathrm{out}}+b_{\mathrm{out}}.

因此

s_{\mathrm{out}}-s_{\mathrm{in}} \le b_{\mathrm{in}}-b_{\mathrm{out}} \le b_{\max}-b_{\min}=B.

其中 $B=\operatorname{range}(b)$ 是层内有效 bias 范围。又因为 $e_{\mathrm{out}}$ 属于 raw-score Top-K，所以 $s_{\mathrm{out}}\ge s_{(K)}$ ，从而

s_{\mathrm{in}}\ge s_{(K)}-B.

这是一个严格的 raw-score 损失上界，但它不能单独限制换入 expert 的 raw rank 或归一化 mixture weight，但是可以方便我们后续诊断分析实际的 checkpoint。主流模型使用 Sigmoid 产生 $s_i$ ，其值在 0 到 1 之间（DeepSeek作为ALF的开山鼻祖，在V4版本主动启用Sigmoid, 使用 $\sqrt{\operatorname{softplus}}$ 计算路由分数，因此其 $\Delta b$ 数值不宜与 Sigmoid 模型直接横比）。

checkpoint 中的 bias 有效范围

对每一个 learned-router 层 $\ell$ ，定义有效 bias 范围

\Delta b_\ell=\max_i b_{\ell,i}-\min_i b_{\ell,i}.

下表直接从一些开源 checkpoint 的 router bias tensor 逐层计算并报告 $\Delta b_\ell$ 。

模型	score 激活	$N$	$K$	$\Delta b$ 中位数	$\Delta b$ 最大值
Moonlight-16B-A3B	sigmoid	64	6	0.19226	0.24463
DeepSeek V3/R1 family	sigmoid	256	8	0.04520	0.22078
GLM 4.5 Air	sigmoid	128	8	0.18620	0.37493
GLM 4.7 Flash	sigmoid	64	4	0.16091	0.26885
GLM 4.5/4.7	sigmoid	160	8	0.26385	0.57069
GLM 4.6	sigmoid	160	8	0.26546	0.57069
GLM 5/5.1/5.2	sigmoid	256	8	0.32037	0.64270
Kimi K2	sigmoid	384	8	0.27525	0.78318
Kimi K2.5	sigmoid	384	8	0.33179	0.77349
Kimi K3	sigmoid	896	16	0.34118	0.84703

表 1：checkpoint 静态审计：逐层 correction-bias 有效范围。

从中位数来看， $\Delta b$ 的值相对来说可以接受；但是 GLM 和 Kimi 的最大值仍然可能产生显著的换入换出效应。值得注意的是，DeepSeek V3 系列的 $\Delta b$ 最大值甚至小于 GLM 和 Kimi 的中位数，其中动力学成因倒是有待勘探。

翻转情况的统计

Moonlight-16B-A3B 具有 64 个 routed experts、Top-6 和 26 个 MoE 层。实验使用 32 条、16 个领域的真实文本进行完整模型前向，不使用随机 hidden state。本文将实际集合 $A=\operatorname{TopK}_{s+b}$ 中有、但 raw-score 集合 $R=\operatorname{TopK}_s$ 中没有的 expert，即 $A\setminus R$ ，称为 incoming expert（换入 expert）。相对于每个真实 hidden state 上的 global raw Top-6，得到如下总体统计：

指标	实测值	分母或解释
token-layer 数	26,312	1,012 tokens $\times$ 26 MoE 层
routed-expert dispatch 数	157,872	26,312 $\times$ Top-6
Top-K 集合改变	14,798（56.24%）	全部 token-layer
换入 expert 数	19,654	平均 0.747 个/token-layer
incoming raw rank $>32$	817（4.16%）	全部 19,654 个 incoming
score 损失 $\ge 0.1$	338（1.72%）	$s_{(K)}-s_i$ ，全部 incoming
mixture weight $<0.01$	304（0.1926%）	全部 dispatch；其中 303 个为 incoming
mixture weight $<0.001$	1（0.00063%）	全部 dispatch；该事件为 incoming

表 2：Moonlight 真实激活的总体翻转与低贡献事件统计。

只看“Top-K 是否改变”会把正常的边界重排序和真正低贡献的翻转混在一起。下面进一步报告换入 expert 的 raw score、raw rank、相对 raw Top-K 边界的 score 损失，以及实际 mixture weight 分布：

指标	p01	p05	p50	p95	最大
incoming raw score	0.0083	0.0414	0.1476	0.9564	0.9832
incoming raw rank	7	7	8	30	64
$s_{(K)}-s_i$	0.0004	0.0023	0.0237	0.0808	0.1899
incoming mixture weight	0.0063	0.0217	0.0628	0.1656	0.1709
raw K/K+1 gap	0.0002	0.0011	0.0198	0.1047	0.4415

表 3：Moonlight 换入 expert 与 raw Top-K 边界的分布统计。

低贡献事件并非均匀分布在所有层。第 3、5 和 26 层具有最高的 weight 小于 0.01 dispatch 比例：

层	Top-K 改变	换入/token	rank p95	weight p01	$w<0.01$
3	58.60%	0.842	39	0.00166	1.515%
5	53.95%	0.754	61	0.00952	0.3458%
26	43.68%	0.623	53	0.00218	2.652%
全部层	56.24%	0.747	30	0.00626	0.1926%

表 4：低贡献尾部最集中的 Moonlight 层。

因此，在当前 Moonlight 样本中，普通翻转是常态，但绝大多数换入 expert 仍具有正常的 mixture contribution；真正接近零贡献的事件是稀少且层集中的尾部。当然，该结论来自 Moonlight 16B-A3B 的真实激活，不能直接外推到更大的 DeepSeek-V3、Kimi K3、GLM-5。

不严谨的数学解释

为什么 ALF 本身没有无条件的概率保证，但在实际 checkpoint 诊断中又没有发现大量接近零贡献的翻转？下面给出两个不算特别严谨的直观解释，权作理解，不太能够深究。

$\mathbf{b}$ 的对称初始化和负反馈更新

$\mathbf{b}$ 的更新规则为：

\mathbf{b}\leftarrow\mathbf{b}-\gamma\operatorname{sign}(\mathbf{F}-\mathbf{Q}).

其中 $\mathbf{F}$ 表示按全部 routed assignments 归一化后的实际负载分配， $\mathbf{Q}=(1/N,1/N,\ldots,1/N)$ 表示理想的负载分配。若 $F_i$ 定义为每个 token 选中 expert i 的概率，则相应目标应写成 $K/N$ 。

这里的 sign 起到了负反馈作用：一旦 $F_i>1/N$ ，就尝试减小 $b_i$ ；一旦 $F_i<1/N$ ，就尝试增加 $b_i$ 。 $\mathbf{b}$ 通常从 $\mathbf{0}$ 对称初始化（在 ALF 的原始 paper 中明确提及）。虽然训练会打破这种对称性，但是从直觉上来看，负反馈机制仍能够在某种程度上避免 $\mathbf{b}$ 偏离对称状态太多。另外，DeepSeek 给出的 $\gamma$ 推荐参数是 $10^{-3}$ ，相比于 Sigmoid 0 到 1 的范围较小，也不至于突然偏离太狠。

考虑负载均衡的平衡条件

假设在 token 总体分布上达到了理想的边际负载均衡。令 $\tau(h)$ 表示 token $h$ 的第 $K$ 大 corrected score，则严格的平衡条件是

\Pr\bigl(s_i(h)+b_i\ge\tau(h)\bigr)\approx\frac{K}{N}.

这里 $\tau(h)$ 依赖该 token 的全部 expert scores，并且与 $s_i(h)$ 相关，若进一步采用 mean-field 近似，假设 $\tau(h)$ 可以用近似常数 $\tau$ 代替，并忽略它与 $s_i$ 的相关性，则有

\Pr(s_i\le\tau-b_i)\approx 1-\frac{K}{N}.

假设 $s_i$ 的分布函数是 $F_i$ ，我们有：

F_i(\tau-b_i)\approx 1-\frac{K}{N}.

整理一下有：

b_i\approx\tau-F_i^{-1}\left(1-\frac{K}{N}\right).

对于 DeepSeek-V3， $K=8,N=256$ ，所以 $1-K/N=0.96875\approx 0.97$ ，从而

b_i\approx\tau-F_i^{-1}(0.97).

如果系统保持理想的负载均衡，可以猜想各个专家的分布函数差距不大，因此 $b_i$ 的差距也不会很大。

Open Questions

如何想办法量化 ALF 方法下，这种 expert 换入换出的影响程度？真的有害吗？
有没有最后 $\Delta b$ 差距不显著更加严谨的数学解释？
如何解释不同厂商的模型 $\Delta b$ 范围的差异？这是因为数据、架构还是训练参数的设定？比如 Kimi 系列、GLM 系列和 DeepSeek 系列各自的 $\Delta b$ 范围相对接近，但是不同厂商的模型差距比较明显；另外， $\Delta b$ 看起来和模型尺寸正相关。

恢复了旧博客！

Tue, 16 Jun 2026 02:57:24 +0800

让codex从html中抽取并恢复了2024年及以前的旧博客，读来深感亲切！

其实这件事情想做已久，但是markdown原稿已经丢失，只有html的版本，所以懒得做。而现在Coding Agent已经足够发达，所以偶然想起，心血来潮，让Codex迅速搞定了！

RL Note 3: Markov Decision Process

Thu, 26 Mar 2026 21:58:38 +0800

Prologue

It’s almost been half a year since I first decided to kick off this RL notes series. I apologize for the delay – I’ve been busy with other work, or honestly, just lacking persistence :( Also I have to say I struggled a bit to write my last post on the topic of MAB since the math is quite tough :(

Anyway, let’s continue the journey of RL with the well-known MDP – Markov Decision Process.

Introduction

Let’s first revisit the concept of state in the first post:

s_t = f(o_0, a_0, r_1, o_1, a_1, r_2, \ldots, o_t)

Define history $h_t = \left(o_0, a_0, r_1, o_1, a_1, r_2, \ldots, o_t\right)$ .

In a general RL process, the agent will take an action $a_t$ based on the full history $h_t$ , and then recevie a new reward $r_{t+1}$ and observe a new $o_{t+1}$ .

However, modeling based on the full history is often intractable. If we can find a function $f$ such that $s_t$ is a sufficient statistic of the history for predicting future, then:

\Pr\left(o_{t+1},r_{t+1}\mid h_t, a_t\right) = \Pr\left(o_{t+1},r_{t+1}\mid s_t, a_t\right)

In this case, $s_t$ is a Markov state.

If $o_t$ itself is already a sufficient statistic of the history (i.e., the system is fully observable), we can simply pick $s_t:=o_t$ , which leads to the Markov Property:

Definition: Markov Property

Assume that the state $s_t$ contains all the useful information in history $h_t$ . Then we have:

\Pr\left(s_{t+1}, r_{t+1} \mid s_t, a_t\right) = \Pr\left(s_{t+1}, r_{t+1} \mid h_t, a_t\right)

One example that satisfies the Markov Property is a board game. It’s enough to decide the next move once we know the current board. We don’t care about how the current state was reached. On the contrary, we cannot predict the direction of a ping-pong ball from a single video frame. We have to know more previous frames, which doesn’t satisfy the standard Markov Property. Hence, the Markov Property is actually a simplification of real scenarios, but it still covers many practical scenarios and makes the math much easier and more elegant.

Markov Decision Process

Perfect! You’ve got the main idea for an MDP now: given the current state and action, the next state and reward do not depend on the earlier history. The final step in defining MDP is to introduce the environment and decision-making process.

Definition: Markov Decision Process

The Markov Decision Process can be represented as a 5-tuple where the states satisfy the Markov Property:

\left(\mathcal S, \mathcal A, P, R, \gamma \right)

s.t.

\Pr\left(s_{t+1},r_{t+1}\mid s_t,a_t\right) = \Pr\left(s_{t+1},r_{t+1}\mid h_t,a_t\right)

$\mathcal S$ : the state space.
$\mathcal A$ : the action space.
$P$ : the transition dynamics, where $P\left(s'\mid s,a\right) = \Pr\left(S_{t+1}=s' \mid S_t = s, A_t = a\right)$ represents the probability of reaching state $s'$ from state $s$ after taking action $a$ .
$R$ : the reward function, where $R\left(s,a,s'\right) = \mathbb E\left[R_{t+1} \mid S_t = s, A_t = a, S_{t+1}=s'\right]$ represents the expected reward obtained when transitioning from state $s$ to state $s'$ after taking action $a$ .
$\gamma$ : the discount factor which belongs to $\left[0,1\right]$ .

For the notation $R$ , it is kind of confusing whether it is part of the MDP or a random variable standing for reward. For our discussion, $R(s,a,s')$ with brackets corresponds to the former, while $R_k$ with a subscript corresponds to the latter.

The Markov Property is exactly what allows us to define $P(s' \mid s,a)$ and $R(s,a,s')$ using only the current state-action pair, instead of the whole history.

Bellman Equation

As we talked about in the first post, the goal of RL is to find the states with highest value. So it’s crucial to consider the values in the MDP.

A key feature of MDPs is their recursive structure, as shown above. Thus, we aim to express the value functions recursively. Before formal discussion, let’s define a new notation to simplify the discussion:

Definition: (Discounted) Return

The (discounted) return is a random variable representing the discounted cumulative reward along a trajectory.

G_t = \sum_{k=t+1}^{T}\gamma^{k-t-1} R_{k}

Here $T$ is the terminal time for an episodic task; for a continuing task, we may take $T=\infty$ .

By the definitions in the first post, we have:

Q\left(s_t,a_t\right) = \mathbb E_{\pi}\left[G_t \mid S_t =s_t,A_t=a_t\right]

\begin{aligned} v\left(s_t\right) &= \mathbb{E}_{a_t\sim\pi\left(\cdot\mid s_t\right)}\left[Q\left(s_t,a_t\right)\right] \\ &= \mathbb{E}_{a_t\sim\pi\left(\cdot\mid s_t\right)}\left[\mathbb E_{\pi}\left[G_t\mid S_t=s_t,A_t=a_t\right]\right] \\ &= \mathbb E_{\pi}\left[G_t\mid S_t=s_t\right] \\ &= \mathbb E_{\pi}\left[R_{t+1} + \gamma G_{t+1}\mid S_t=s_t\right] \\ &= \mathbb E_{\pi}\left[R_{t+1} \mid S_t=s_t\right] + \gamma \mathbb E_{\pi}\left[G_{t+1} \mid S_t=s_t\right] \\ \end{aligned}

For the first half, we have:

\begin{aligned} \mathbb{E}_\pi\left[R_{t+1}\mid S_t=s_t\right] &= \sum\limits_{a_t}\pi\left(a_t\mid s_t\right)\sum_{s_{t+1}}\mathbb E\left[R_{t+1}\mid S_t = s_t, A_t = a_t, S_{t+1} = s_{t+1} \right]\Pr\left(S_{t+1}=s_{t+1}\mid S_t =s_t, A_t =a_t\right) \\ &= \sum\limits_{a_t}\pi\left(a_t\mid s_t\right)\sum_{s_{t+1}}R\left(s_t,a_t,s_{t+1}\right)P\left(s_{t+1}\mid s_t,a_t\right) \\ \end{aligned}

For the second half, we have:

\begin{aligned} \mathbb E_{\pi}\left[G_{t+1} \mid S_t=s_t\right] &= \mathbb E_\pi\left[\mathbb E_{\pi}\left[G_{t+1}\mid S_{t+1} \right]\mid S_t=s_t\right] \\ & = \mathbb E_\pi\left[v\left(S_{t+1}\right)\mid S_t = s_t\right] \\ & = \sum_{a_{t}}\pi\left(a_t\mid s_t\right)\sum_{s_{t+1}}\Pr\left(S_{t+1} = s_{t+1}\mid S_t =s_t, A_t = a_t\right)v\left(s_{t+1}\right) \\ & = \sum_{a_{t}}\pi\left(a_t\mid s_t\right)\sum_{s_{t+1}}P\left(s_{t+1}\mid s_t,a_t\right)v\left(s_{t+1}\right) \end{aligned}

Combine the two halves together, we have:

v\left(s_t\right) = \sum_{a_t}\pi\left(a_t\mid s_t\right)\sum_{s_{t+1}}P\left(s_{t+1}\mid s_t,a_t\right)\left[R\left(s_t,a_t,s_{t+1}\right)+\gamma v\left(s_{t+1}\right)\right]

Remove all notations with time subscripts, we have the Bellman Equation.

Definition: Bellman Equation

v\left(s\right) = \sum_{a}\pi\left(a\mid s\right)\sum_{s'}P\left(s'\mid s,a\right)\left[R\left(s,a,s'\right)+\gamma v\left(s'\right)\right]

Let’s take an example of the Bellman Equation to better understand it.

Example: Bellman Equation

Cells with number have given values. The arrows on the left indicate the reward for each move. Hitting a wall yields a reward of $-1$ and the agent remains in the same position. All other moves yield zero rewards. Actions are chosen uniformly at random. $\gamma = 0.9$ .

Compute the values for the two highlighted cells on the right.

It’s easy to see the $P(\cdot)$ are one-hot and all $\pi(a\mid s)=0.25$ . By the Bellman Equation, we have:

v(A) = 0.25\cdot 0.9\cdot(2.3+0.7-0.4+0.4) = 0.675 \approx 0.7

v(B) = 0.25\cdot 0.9\cdot(-1.2-1.4) + 0.25\cdot 2\cdot (-1+0.9v(B))

Solving it, we have $v(B) \approx -2.0$ .

Bellman Equation In a Matrix Form

Let’s rewrite the Bellman Equation in a matrix form to make it more unified and elegant. This is feasible in a tabular environment.

Let’s define:

r_\pi\left(s\right) = \sum_{a}\pi\left(a\mid s\right)\sum_{s'}P\left(s'\mid s,a\right)R\left(s,a,s'\right)

And suppose $\mathcal S$ and $\mathcal A$ are finite:

\mathcal S = \left\{s_i\right\}_{i=1}^n

\mathcal A = \left\{a_i\right\}_{i=1}^m

Then we can define vectors:

\mathbf v_\pi \in \mathbb R^n,\quad\left(\mathbf v_\pi\right)_i = v_\pi(s_i)

\mathbf r_\pi \in \mathbb R^n,\quad\left(\mathbf r_\pi\right)_i = r_\pi(s_i)

And the transition matrix:

\mathbf P_\pi\in\mathbb R^{n\times n},\quad \left(\mathbf P_\pi\right)_{ij} = \sum_a\pi\left(a\mid s_i\right)P\left(s_j\mid s_i,a\right)

Then we can define the Bellman Equation in a matrix form:

\mathbf v_\pi = \mathbf r_\pi + \gamma\mathbf P_\pi \mathbf v_\pi

Rearranging it, we have:

\left(\mathbf I-\gamma \mathbf P_\pi\right)\mathbf v_\pi = \mathbf r_\pi

To make the solution on the right well-defined, we need to show that $\mathbf I-\gamma \mathbf P_\pi$ is invertible.

Proof: Invertibility of

\mathbf{I}-\gamma\mathbf{P}_{\pi}

Since $\mathbf P_\pi$ is the probability matrix, we have

\mathbf P_\pi = \left(\mathbf p_1,\mathbf p_2,\ldots,\mathbf p_n\right)^\top

and

\forall i\in\left[1,n\right], \sum_j \left(\mathbf p_i\right)_j = 1, \left(\mathbf p_i\right)_j \ge 0

Then

\left|\mathbf p_i^\top\mathbf v\right| = \left|\sum_j \left(\mathbf p_i\right)_j\left(\mathbf v\right)_j\right| \le \sum_j \left(\mathbf p_i\right)_j\left|\left(\mathbf v\right)_j\right|\le \max_j\left|\left(\mathbf v\right)_j\right| = \left\|\mathbf v\right\|_\infty

Then

\left\|\mathbf P_\pi\mathbf v\right\|_\infty = \max_i\left|\mathbf p_i^\top \mathbf v\right| \le \left\|\mathbf v\right\|_\infty

Then

\begin{aligned} \left\|\left(\mathbf{I}-\gamma\mathbf{P}_{\pi}\right)\mathbf v \right\|_\infty&=\left\|\mathbf v-\gamma\mathbf P_\pi\mathbf v \right\|_\infty \\ &\ge \left\|\mathbf v\right\|_\infty - \gamma\left\|\mathbf P_\pi\mathbf v\right\|_\infty \\ & \ge \left(1-\gamma\right)\left\|\mathbf v\right\|_\infty \end{aligned}

Then when $\gamma < 1$ and $\mathbf v\ne \mathbf 0$ , $\left\|\left(\mathbf{I}-\gamma\mathbf{P}_{\pi}\right)\mathbf v \right\|_\infty>0$ holds, which means $\mathbf{I}-\gamma\mathbf{P}_{\pi}$ is invertible.

Hence, when $\gamma<1$ , we can express $\mathbf v$ as

\mathbf v_\pi = \left(\mathbf I-\gamma \mathbf P_\pi\right)^{-1}\mathbf r_\pi

Conclusions

Congratulations! We’ve covered the most fundamental modeling framework in RL—MDP. In most cases, we will assume that the environment satisfies the MDP assumptions.

It’s necessary to clarify that the matrix-form discussion in this section applies to a finite, or tabular, MDP. We assume that both the state space and the action space are finite and discrete, which allows us to represent the problem using linear algebra.

Takeaways

The main idea for Markov Decision Process is to assume the Markov Property: $\Pr\left(s_{t+1}, r_{t+1} \mid s_t, a_t\right) = \Pr\left(s_{t+1}, r_{t+1} \mid h_t, a_t\right)$
Bellman Equation gives a recursive characterization of the value function in an MDP. $v\left(s\right) = \sum_{a}\pi\left(a\mid s\right)\sum_{s'}P\left(s'\mid s,a\right)\left[R\left(s,a,s'\right)+\gamma v\left(s'\right)\right]$
We can express the Bellman Equation in a matrix form to make it more unified and elegant. $\mathbf v_\pi = \left(\mathbf I-\gamma \mathbf P_\pi\right)^{-1}\mathbf r_\pi$

In the next post, we will go deeper into a direct utilization of MDP: dynamic programming (DP), which I believe most of you are familiar with from your first programming class at university. However, we will view it in a more RL way, and I believe you may gain a new understanding of it.

RL Note 2: Multi-Armed Bandits

Mon, 03 Nov 2025 21:11:10 +0800

Prologue

In the last post, we introduced the basics of RL—action, reward, state, value, policy, model, etc.—so you should now have a rough picture of the field. In this post, we go deeper and discuss a classic yet still active topic: Multi-Armed Bandits (MAB). There will be more math ahead; hope you can enjoy it.

Problem Formulation

Multi-armed bandits are popular gambling games. There are slot machines, each called a bandit, with an unknown reward distribution that governs how much you get when pulling its arm. Your goal is to maximize the total winnings within a fixed number of pulls. You may try the game to get an intuition at https://su-my.github.io/Test-page.

Let’s abstract the gambling game to a formal problem definition:

Definition: Stochastic Multi-Armed Bandit(MAB) Problem

Let

\mathcal{A} = \{ a_1, a_2, \dots, a_K \}

be the set of arms. Pulling arm

a_k

at round

t

yields a reward

R_{t,k} \sim \mathcal{D}_k

, where each

\mathcal{D}_k

is an unknown distribution with mean

\mu_k

. Over a finite horizon

T

, a policy chooses arms

(a_{i_1}, \dots, a_{i_T})

and observes rewards

(R_{1,i_1}, \dots, R_{T,i_T})

. The objective is to maximize the expected cumulative reward

\mathbb{E}_\pi\left[\sum_{t=1}^{T} R_{t,i_t}\right]

where the $R_i$ notations are random variabes for rewards.

Furthur discussions:

Regret

Definitions

Regret is the primary metric for evaluating a MAB algorithm. Informally, it is the gap between the reward you would obtain by always pulling the best arm and the reward actually obtained by the algorithm. A smaller regret indicates a better algorithm.

Throughout this section we assume a stochastic environment with arm-wise reward means $\{\mu_k\}_{k=1}^K$ , and we write $\mu^* = \max\limits_{1\le k\le K} \mu_k$ for the optimal mean.

Definition: Realized Pseudo-Regret

Fix a policy $\pi$ and a sample path $\omega=(a_{i_1}, a_{i_2},\dots,a_{i_T})\in\Omega$ .
Let $A_t:\Omega\to\mathcal{A}$ denote the random arm chosen at round $t$ , and write $A_t(\omega)=a_{i_t}$ for its realization along $\omega$ .
The realized pseudo-regret at horizon $T$ is

\mathbf R_\pi(T,\omega) = \sum_{t=1}^T \bigl(\mu^* - \mu(A_t(\omega))\bigr) = T\mu^* - \sum_{t=1}^T \mu_{i_t}.

Definition: Random Pseudo-Regret

Under policy $\pi$ , the (trajectory-dependent) random pseudo-regret is the random variable

\mathbf R_\pi(T) = \sum_{t=1}^T \bigl(\mu^* - \mu(A_t)\bigr) = T\mu^* -\sum_{t=1}^T\mu(A_t),

It represents the regret as a random variable before taking expectation.

Definition: Expected Pseudo-Regret

The expected pseudo-regret of policy $\pi$ is

R_{\pi}(T) = \mathbb{E}_\pi[\mathbf R_\pi(T)] = \int_\Omega \mathbf R_\pi(T,\omega)\,d\mathbb{P}_\pi(\omega),

which measures the algorithm’s average performance under its induced randomness.

The three regret notions above may look verbose, but they separate measurables cleanly and avoid mixing pathwise quantities with expectations¹. Two remarks:

Why pseudo-regret?
Pseudo‑regret replaces realized rewards by their means. It isolates the algorithm’s decision quality from observation noise. A corresponding “real” (pathwise) regret can be defined by using realized rewards. Let $a^*\in\arg\max_k \mu_k$ and let $r_t(a)$ denote the realized reward at round $t$ if arm $a$ were pulled. Then the realized regret along $\omega$ is
$\mathbf R^{\mathrm{real}}_\pi(T,\omega) = \sum_{t=1}^T \bigl(r_t(a^*) - r_t(A_t(\omega))\bigr).$
Taking expectation over the reward noise (with $r_t(a)\sim\mathcal D_a$ ) recovers the pseudo‑regret:
$\mathbb E\bigl[\mathbf R^{\mathrm{real}}_\pi(T,\omega)\mid A_1(\omega),\dots,A_T(\omega)\bigr] = \mathbf R_\pi(T,\omega).$
In analysis we usually work with pseudo‑regret, and when context is clear we simply say “regret”.
Why is $A_t$ random?
Because both the rewards and the policy may be stochastic. Rewards influence the history observed by the policy, and the policy may randomize given that history; therefore $A_t:\Omega\to\mathcal A$ is a random variable.

Lower Bounds

For convenience, we only talk about the realized regret under a fixed trajectory $\omega$ and a fixed policy $\pi$ and simplify the $\mathbf {R}_\pi(T,\omega)$ as $\mathbf R(T)$ .

As mentioned before, we should make the regret grow slower. So what is the lower bound for regret?

It’s easy to find out any algorithm cannot be worse than linear, since:

\begin{aligned} \mathbf R(T) &= \sum_{t=1}^T\left(\mu^*-\mu(A_t(\omega))\right) \\ & \leq \sum_{t=1}^T\left(\mu^*-\mu'\right) \\ & = T\Delta \\ &= \Omega(T) \end{aligned}

where $\mu' = \min\limits_{1\le k\le K} \mu_k, \Delta = \mu^*-\mu'$ .

Hence, a wise algorithm should be sub-linear, i.e. $o(T)$ . Two common lower bounds are:

Gap-independent:
$\Omega(\sqrt{TK})$
Gap-dependent:
$\Omega\left(\sum_{a_i \ne a^*} \left( \frac{\Delta_i}{KL(P_{a_i}, P_{a^*})} \right) \log T\right)$

where $K$ is the number of arms; $\Delta_i$ is the gap between the mean of $a_*$ and $a_i$ ; the gap-independent means it’s hard to identify the gap between the arms; gap-dependent is the opposite.

Intuitively, the easier the gap is to indentify, the less attention will be paid to find out the best arm and we will get lower regret. That’s why the gap-dependent lower bound is $\Omega(\log T)$ which is less than $\Omega(\sqrt{T})$

Proofs will be provided once I figured them out.

Explore-Exploit Dilemma

The difficulty of MAB comes from the explore-exploit dilemma, which is intuitive once you get to know what MAB problems are chasing for.

Explore: we must pay some steps to explore the best policy for choosing among arms to avoid commiting to the wrong arms causing linear regrets later. But the exploration phase is always along with regret accumulation itself because of the wrong attempts we must meet.
Exploit: Stick to current policy. As talked before, this may leads to high regret if the exploration is not sufficient.

MAB From the Perspective of RL

As the broad picture you may have for MAB now, it’s just a RL-like problem with simplified environments. You may treat the reward in MAB as a combination of observation and reward in RL.

Tail Bounds²

To get into the real discussion of MAB, we first state a few mathematical propositions that describe how far samples of a random variable can deviate from its expectation. This is necessary for further analysis of MAB algorithms since we are estimating the means $\mu_k$ ; if we can estimate these means accurately and quickly, we can obtain low regret.

If you are not interested in the math details, you can jump to Hoeffding’s Inequality and skip this section omitting all proofs.

Markov’s Inequality

Proposition: Markov’s Inequality

\Pr(X\ge a)\le \frac{\mathbb E(X)}{a}

where $X$ is a non-negative random variable.

Proof: Markov’s Inequality

Consider the indicator function for $X\ge a$ :

\mathbf 1_{\{X\ge a\}} = \begin{cases} 1 & X\ge a \\ 0 & X < a \end{cases}

By the definition of expectation, $\mathbb E\bigl[\mathbf 1_{\{X\ge a\}}\bigr] = \Pr(X\ge a)$ .

If $X < a$ , then $\mathbf 1_{\{X\ge a\}}=0$ and $X\ge 0 = a\,\mathbf 1_{\{X\ge a\}}$ ; if $X\ge a$ , then $\mathbf 1_{\{X\ge a\}}=1$ and $X\ge a = a\,\mathbf 1_{\{X\ge a\}}$ . Thus $X\ge a\,\mathbf 1_{\{X\ge a\}}$ always holds.

Taking expectations on both sides yields

\mathbb E(X)\ge a\,\mathbb E\bigl[\mathbf 1_{\{X\ge a\}}\bigr] = a\,\Pr(X\ge a),

i.e.

\Pr(X\ge a)\le \frac{\mathbb E(X)}{a}.

$\square$

Chebyshev’s Inequality

Proposition: Chebyshev’s Inequality

\Pr(\lvert X-\mu\rvert\ge k\sigma)\le\frac{1}{k^2},\quad \text{for any } k>0,

where $\mu$ is the expectation of $X$ , and $\sigma$ is the standard deviation of $X$ .

Proof: Chebyshev’s Inequality

Using Markov’s inequality, we have:

\begin{aligned} \Pr(\lvert X-\mu\rvert\ge k\sigma) & = \Pr(\lvert X-\mu\rvert^2\ge k^2\sigma^2)\\ & \le\frac{\mathbb E\bigl[(X-\mu)^2\bigr]}{k^2\sigma^2} \\ & = \frac{1}{k^2}, \end{aligned}

where the last equality uses $\operatorname{Var}(X)=\mathbb E\bigl[(X-\mu)^2\bigr]=\sigma^2$ . $\square$

Chernoff Bound

The Chernoff bound is a technique to estimate tail bounds quantitatively, i.e., $\Pr(X\ge a)$ or $\Pr(X\le a)$ . We can summarize it in three steps:

Chernoff Bound Trick Steps

For $t\in\mathbb R$ , apply the map $x\mapsto e^{tx}$ to obtain $\Pr(e^{tX}\ge e^{ta})$ . When $t>0$ this bounds the upper tail $\Pr(X\ge a)$ ; when $t<0$ it bounds the lower tail $\Pr(X\le a)$ . This ensures the random variable $e^{tX}$ is non‑negative so Markov’s inequality applies.
Apply Markov’s inequality: $\Pr(e^{tX}\ge e^{ta})\le e^{-ta}\,\mathbb E\bigl[e^{tX}\bigr].$
Choose $t$ to make the right‑hand side as small as possible (optimize over $t$ ): $\Pr(X\ge a)\le \inf\limits_{t>0} e^{-ta} M_X(t),\quad \Pr(X\le a)\le \inf\limits_{t<0} e^{-ta} M_X(t),$ where $M_X(t)=\mathbb E\bigl[e^{tX}\bigr]$ is the moment generating function (MGF) of $X$ .

As an illustration, consider Bernoulli trials and apply the Chernoff bound.

Suppose we have $n$ i.i.d. Bernoulli trials $X_i$ with success probability $p$ . Let $X=\sum_i X_i$ ; then $\mu = \mathbb E[X]=np$ . Suppose $\lambda > 1$ .

\begin{aligned} \Pr(X\ge \lambda\mu)&\le \inf\limits_{t>0} e^{-t\lambda np}\,\mathbb E\bigl[e^{tX}\bigr]\\ &=\inf\limits_{t>0} e^{-t\lambda np}\, \prod_{i=1}^n \mathbb E\bigl[e^{tX_i}\bigr]\\ &=\inf\limits_{t>0} e^{-t\lambda np}\, (pe^t+1-p)^n \\ &=\inf\limits_{t>0} \exp\bigl(n\log (pe^t+1-p)-t\lambda np\bigr) \end{aligned}

Consider the function $f(t) = n\log (pe^t+1-p)-t\lambda np$ . Taking the derivative and setting it to $0$ , we have

f'(t) = \frac{npe^t}{pe^t+1-p}-\lambda np = 0

i.e.,

t^*=\log \lambda+\log(1-p)-\log(1-\lambda p).

It is easy to verify that $f(t)$ attains its minimum at $t^*$ . Substituting $t^*$ into $f(t)$ , we have

f(t^*) = -n\left((1-\lambda p)\log \frac{1-\lambda p}{1-p}+\lambda p \log \lambda\right) = -n\,D_{\text{KL}}(\lambda p \parallel p),

where $D_{\text{KL}}(q\parallel p)= q\log\!\frac{q}{p} + (1-q)\log\!\frac{1-q}{1-p}$ is the Kullback–Leibler divergence between Bernoulli parameters.

$t^* > 0$ since $\lambda > 1$ . Hence, we have

\Pr(X\ge \lambda\mu)\le \exp\bigl(-n\,D_{\text{KL}}(\lambda p \parallel p)\bigr).

Hoeffding’s Inequality

The protagonist finally comes! The Hoeffding’s inequality are bounding a set of independent bounded random variables. You may find it especially useful in MAB problems since the arms are basically a set of independent random variables.

Hoeffding’s Inequality is actually a special case of Chernoff Bound. Let’s give the proposition first:

Proposition: Hoeffding’s Inequality

Given a set of independent bounded random varibales $\left\{X_i\right\}_{i=1}^n$ with $X_i\in[a_i,b_i]$ . Let $X=\sum_i X_i$ and $\mu=\mathbb E[X]$ . Then

\Pr(X-\mu\ge \varepsilon)\le \exp\left(-\frac{2\varepsilon^2}{\sum_{i=1}^n (b_i-a_i)^2}\right)

\Pr(X-\mu\le -\varepsilon)\le \exp\left(-\frac{2\varepsilon^2}{\sum_{i=1}^n (b_i-a_i)^2}\right)

It’s easy to find out the Hoeffding’s inequality is a special case of Chernoff Bound.

Proof: Hoeffding’s Inequality

Let $\mu_i = \mathbb E[X_i]$ , $\varepsilon > 0$ . Then

\begin{aligned} \Pr(X-\mu\ge \varepsilon)&\le \inf\limits_{t>0} e^{-t\varepsilon}\,\mathbb E\left[e^{t(X-\mu)}\right]\\ &=\inf\limits_{t>0} e^{-t\varepsilon}\, \mathbb E\left[e^{t\sum_{i=1}^n(X_i-\mu_i)}\right]\\ &=\inf\limits_{t>0} e^{-t\varepsilon}\, \prod_{i=1}^n \mathbb E\left[e^{t(X_i-\mu_i)}\right] \end{aligned}

According to Hoeffding’s Lemma, we have

\mathbb E\left[e^{t(X_i-\mu_i)}\right]\le \exp\left(\frac{t^2(b_i-a_i)^2}{8}\right)

Hence

\begin{aligned} \Pr(X-\mu\ge \varepsilon)&\le \inf\limits_{t>0} e^{-t\varepsilon}\, \prod_{i=1}^n \exp\left(\frac{t^2(b_i-a_i)^2}{8}\right)\\ &=\inf\limits_{t>0} \exp\left(-t\varepsilon+\frac{t^2}{8}\sum_{i=1}^n (b_i-a_i)^2\right) \\ &=\exp\left(-\frac{\varepsilon^2}{2\sum_{i=1}^n (b_i-a_i)^2}\right) \end{aligned}

The last equality is taken when $t=\frac{4\varepsilon}{\sum_{i=1}^n (b_i-a_i)^2}$ $\square$

We used Hoeffding’s Lemma to bound the MGF of $X_i-\mu_i$ :

Hoeffding’s Lemma

Let $X\in[a,b]$ and $\mu=\mathbb E[X]$ . Then $\forall t \in \mathbb R$

\mathbb E\left[e^{t(X-\mu)}\right]\le \exp\left(\frac{t^2(b-a)^2}{8}\right)

Proof: Hoeffding’s Lemma

Consider Y = $X-\mu$ , then $\mathbb E[Y] = 0$ Since $e^{tY}$ is convex, we have:

e^{tY} = \frac{b-Y}{b-a}e^{ta} + \frac{Y-a}{b-a}e^{tb}

Take expectation on both sides, we have

\mathbb E\left[e^{tY}\right] = \frac{b}{b-a}e^{ta} - \frac{a}{b-a}e^{tb}

Let $u=t(b-a)$ and $\lambda = \frac{-a}{b-a}$ , then

\mathbb E\left[e^{tY}\right] = e^{-\lambda u} \left(1-\lambda + \lambda e^{u}\right) = \exp\left(-\lambda u + \log \left(1-\lambda+\lambda e^{u}\right)\right)

Consider $g(u)=-\lambda u + \log \left(1-\lambda+\lambda e^{u}\right)$ , it’s easy to verify $g(0) = g'(0) = 0$ .

And $g''(u) = \frac{(1-\lambda)\lambda e^u}{(1-\lambda+\lambda e^{u})^2}\le\frac{1}{4}$ R_{\operatorname{ETC}(m)} (n) \le m\sum_{i=2}^k\Delta_i + (n-mk)\sum_{i=2}^k\exp\left(-\frac{m\Delta_i^2}{4}\right)

Hence $\exist \xi\in(0,u)$ s.t. $g(u) = g(0) + g'(0)u + \frac{1}{2}g''(\xi)u^2 \le \frac{u^2}{8}$

Which means $\mathbb E\left[e^{tY}\right]\le \exp\left(\frac{u^2}{8}\right)$

$\square$

Let’s also give a special case of Hoeffding’s Inequality here when all random variables are i.i.d. and sub-gaussian.

Hoeffding’s Inequality for i.i.d. sub-gaussian random variables

Let $X_1,\ldots,X_n$ be i.i.d. sub-gaussian random variables with sub-gaussian norm $\sigma$ , which means

\mathbb E\left[e^{t(X_i-\mu)}\right]\le \exp\left(\frac{t^2\sigma^2}{2}\right)

Let $\bar X=\frac{1}{n}\sum_{i=1}^n X_i$ and $\mu=\mathbb E[\bar X]$ . Then

\Pr(\lvert\bar X-\mu\rvert\ge \varepsilon)\le 2\exp\left(-\frac{n\varepsilon^2}{2\sigma^2}\right)

Proof: Hoeffding’s Inequality for i.i.d. sub-gaussian random variables

The both sides are symmetric. Let’s prove the case $\Pr(\bar X-\mu\ge \varepsilon)$ .

It’s easy to verify that $\bar X$ is also sub-gaussian with sub-gaussian norm $\frac{\sigma}{\sqrt{n}}$ and .

\begin{aligned} \Pr(\bar X -\mu\ge\varepsilon) &= \Pr(e^{t(\bar X-\mu)}\ge e^{t\varepsilon}) \\ & = \inf\limits_{t>0} e^{-t\varepsilon}\,\mathbb E\left[e^{t(\bar X-\mu)}\right] \\ & \le \inf\limits_{t>0} \exp\left(-t\varepsilon + \frac{t^2\sigma^2}{2n}\right) \\ & = \exp\left(-\frac{n\varepsilon^2}{2\sigma^2}\right) \end{aligned}

Explore-Then-Commit

Explore-Then-Commit (ETC) is an intuitive algorithm for MAB problems. It pulls each arm a fixed number of times, then commits to the arm with the highest estimated mean.

Definition: Explore-Then-Commit Algorithm

Given the following environment:

$k$ : number of arms
$n$ : horizon, $n > k$
$\mathcal A = \{a_i\}_{i=1}^{k}$

The arm chosen at round $t$ is:

A_t = \begin{cases} (t\mod k) + 1 & t\le mk \\ \argmax_{a\in\mathcal A} \hat \mu_a & t > mk \end{cases}

where $m<\lfloor\frac{n}{k}\rfloor$ is the number of times each arm is pulled and $\hat \mu_a=\frac{1}{m}\sum_{i=1}^m r_{a,i}$ , with $r_{a,i}$ the reward of arm $a$ on its $i$ -th pull. Tie-breaking of $\argmax$ is usually random.

Consider the special case where all arms are $1$ -sub-Gaussian; this helps build intuition for the regret. For brevity, we denote the expected pseudo-regret of ETC at horizon $n$ by $R_{\mathrm{ETC}}(n)$ .

W.l.o.g., assume $\forall i < j, \mu_i \ge \mu_j$ , and define $\Delta_i = \mu_1 - \mu_i$ .

\begin{aligned} R_{\operatorname{ETC}(m)} (n) & = \mathbb E\left[\sum\limits_{i=1}^n \Delta_{A_i}\right] \\ & = \sum\limits_{i=1}^n \sum_{j=1}^k\Pr(A_i=j)\Delta_j \\ & = m\sum_{i=2}^k\Delta_i + (n-mk)\sum_{i=2}^k\Pr(A_{mk+1}=i)\Delta_i \end{aligned}

Moreover,

\begin{aligned} \Pr(A_{mk+1}=i) & \le \Pr(\hat \mu_{i} \ge \max_{a\in\mathcal A, a\ne i}\mu_a) \\ & \le \Pr(\hat \mu_{i} \ge \mu_1) \\ & = \Pr((\hat\mu_{i}-\hat\mu_1) - (\mu_i-\mu_1)\ge \Delta_i) \\ & \le \exp\left(\frac{-m\Delta_i^2}{4}\right) \end{aligned}

The first inequality is strict when multiple arms tie, because the tie-breaking rule is random. The last inequality follows since $X-Y$ is $\sqrt{2}$ -sub-Gaussian when $X$ and $Y$ are $1$ -sub-Gaussian, and then we can apply the Hoeffding’s Inequality.

Hence, we have

R_{\operatorname{ETC}(m)} (n) \le m\sum_{i=2}^k\Delta_i + (n-mk)\sum_{i=2}^k\Delta_i\exp\left(-\frac{m\Delta_i^2}{4}\right)

When $k=2$ and writing $\Delta = \Delta_2$ , we have

R_{\operatorname{ETC}(m)} (n) \le m\Delta + (n-mk)\Delta\exp\left(-\frac{m\Delta^2}{4}\right)\le m\Delta + n\Delta\exp\left(-\frac{m\Delta^2}{4}\right)

Differentiating the RHS with respect to $m$ shows it is minimized at

m=\max\left(1,\left\lceil\frac{4}{\Delta^2}\log\left(\frac{n\Delta^2}{4}\right)\right\rceil\right),

which implies

\boxed{ R_{\operatorname{ETC}(m)} (n) \le \min \left( n\Delta, \Delta + \frac{4}{\Delta} \left( 1 + \max \left( 0, \log \left( \frac{n\Delta^2}{4} \right) \right) \right) \right) }

Note that $m \ge 1$ , and $R_{\operatorname{ETC}(m)} (n)$ cannot exceed $n\Delta$ as discussed above. Informally, this bound says that, for fixed gap $\Delta$ , the regret of ETC grows only logarithmically with the horizon $n$ , but with a relatively large constant and the need to know (or tune around) $\Delta$ to choose $m$ .

We can plot how $R_{\operatorname{ETC}} (n)$ changes with $\Delta$ using a short Python snippet:

Python Code: ETC Regret Upper Bound

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


import numpy as np
import matplotlib.pyplot as plt

n = 1000
eps = 1e-6
deltas = np.linspace(eps, 100.0, 1000)

term_linear = n * deltas
term_log = deltas + (4 / deltas) * (1 + np.maximum(0, np.log(n * deltas**2 / 4)))
bound = np.minimum(term_linear, term_log)

plt.figure(figsize=(6, 4))
plt.plot(deltas, bound, label=r"$R_{\mathrm{ETC}}(n)$ upper bound")
plt.xlabel(r"$\Delta$")
plt.ylabel(r"regret bound")
plt.title(f"ETC regret bound vs Δ (n={n})")
plt.grid(True, ls="--", alpha=0.5)
plt.legend()
plt.tight_layout()
plt.show()

The regret bound diverges as $\Delta \to \infty$ . If we restrict to $\Delta \le 1$ (arms are $1$ -sub-Gaussian), the bound is maximized near $\Delta \approx \frac{2\sqrt{e}}{\sqrt{n}}$ , where $R_{\operatorname{ETC}} (n) \approx \frac{4\sqrt{n}}{\sqrt{e}}$ (from differentiating the RHS with respect to $m$ ). For fixed $\Delta$ , the gap-dependent upper bound scales as $O((\log n)/\Delta)$ , but unlike UCB it needs a good estimate of $\Delta$ (to choose $m$ ) and carries larger constants.

Varaints of ETC

ETC is intuitive but too simple, which leads to the folloing disadvatages:

It requires the prior of $\Delta$ to find the best $m$ for exploration, which are usually unknown in real scenerios.
It is too rigid. It may continually commit to the wrong arms in the exploration phase.
Theorectically, it is usually two times of the gap-dependent lower bound.

Proposition: Lower Bound of ETC

The lower bound of ETC is no less than two times of the gap-dependent lower bound.

Proof: Lower Bound of ETC

We only prove a special case with two arms $a_1$ and $a_2$ with Gaussian distribution $\sigma_1$ . Consider the gap-dependent lower bound:

L_{\text{gap-dependent}} (n) \ge \frac{\Delta\log n}{\operatorname{KL}(P_{a_1}, P_{a_2})}

And

\begin{aligned} \operatorname{KL}\left(\mathcal N(\mu_1, 1), \mathcal N(\mu_2, 1)\right) &= \mathbb E_{x\sim P_{a_1}}\left[\log\left(\frac{P_{a_1}(x)}{P_{a_2}(x)}\right)\right] \\ &=\mathbb E_{x\sim P_{a_1}}\left[\log\left(\frac{\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{(x-\mu_1)^2}{2}\right)}{\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{(x-\mu_2)^2}{2}\right)}\right)\right] \\ &=\left(\mu_1-\mu_2\right)\mathbb E_{x\sim P_{a_1}}\left[x-\frac{\mu_1+\mu_2}{2}\right] \\ & = \frac{\left(\mu_1-\mu_2\right)^2}{2} \\ & = \frac{\Delta^2}{2} \end{aligned}

Hence,

L_{\text{gap-dependent}} (n) = \frac{2}{\Delta} \log n

But the The lower bound of ETC is

L_{\text{ETC}} = \Omega\left(\frac{4}{\Delta}\log n\right)

with a multiplicative constant of $2$ .

$\square$

Hence, there are two common variants of ETC to address the above disadvantages:

Epoch Greedy ETC

Rather than committing to the arm with the highest estimated mean, Epoch Greedy ETC commits to the arm with the highest estimated mean in each epoch.

$\varepsilon$ -Greedy ETC

Toss a coin with probability $\varepsilon$ to commit to the arm with the highest estimated mean, otherwise commit to a random arm.

Upper Confidence Bound (UCB)

UCB is a classical algorithm for MAB problems. It follows the optimism‑in‑the‑face‑of‑uncertainty principle: arms with larger statistical uncertainty receive a positive bonus, so rarely sampled arms are explored more.

Throughout this subsection we write $R_{\mathrm{UCB}}(n)$ for the expected pseudo-regret of the UCB algorithm at horizon $n$ .

Definition: Upper Confidence Bound Algorithm

Given the environment below

$k$ : number of arms
$n$ : horizon, $n > k$
$\mathcal A = \{a_i\}_{i=1}^{k}$

Let $\hat \mu_a$ be the empirical mean of arm $a$ after $N_t(a)$ pulls by round $t$ . The arm chosen at round $t$ is

A_t = \argmax_{a\in\mathcal A} \bigl(\hat \mu_a + B_t(a)\bigr), \qquad B_t(a) = \sqrt{\frac{\alpha\log t}{N_t(a)}},

where $\alpha>0$ is a hyperparameter (commonly $\alpha=2$ ).

The bonus $B_t(a)$ shrinks as $N_t(a)$ grows and increases slowly with $t$ , encouraging exploration of under‑sampled arms.

The form of $B_t(a)$ comes directly from Hoeffding’s Inequality.

Derivation of

B_t(a)

Let arm $a$ have $\sigma$ ‑sub-Gaussian rewards. Hoeffding gives

\Pr\bigl(\hat \mu_a - \mu_a \ge \varepsilon\bigr) \le \exp\!\left(-\frac{N_t(a)\varepsilon^2}{2\sigma^2}\right).

To make this error probability decay at most on the order of $t^{-\beta}$ for some $\beta>0$ , it suffices that

\exp\!\left(-\frac{N_t(a)\varepsilon^2}{2\sigma^2}\right) \le t^{-\beta}.

Solving for $\varepsilon$ suggests choosing

\varepsilon = \sqrt{\frac{\alpha\log t}{N_t(a)}}, \qquad \alpha = 2\sigma^2\beta.

Next we sketch the regret of UCB. Assume all arms are $\tfrac{1}{2}$ ‑sub-Gaussian, and $\beta = 4$ , hence $\alpha = 2$ . Order the arms so that $\mu_1 \ge \mu_2 \ge \cdots \ge \mu_k$ and define the gaps $\Delta_i = \mu_1 - \mu_i$ for $i\ge2$ .

For each round $t$ , define the good event

G_t = \left\{\forall i\in[1,k],\; \bigl|\hat \mu_i - \mu_i\bigr| \le B_t(i)\right\}.

By Hoeffding’s inequality and a union bound over arms, one can choose the constant in $B_t(i)$ (equivalently, in $\beta$ ) so that

\Pr(\neg G_t) = O\!\left(\frac{1}{t^4}\right).

We decompose the regret horizon‑wise:

R_{\operatorname{UCB}}(n) = \sum_{t=1}^n \mathbb E[\Delta_{A_t}] = \sum_{t=1}^n \mathbb E[\Delta_{A_t}\mathbf 1_{G_t}] + \sum_{t=1}^n \mathbb E[\Delta_{A_t}\mathbf 1_{\neg G_t}].

Since the instantaneous regret is at most $\Delta_{\max}\coloneqq\max_{i\ge2}\Delta_i$ , the second sum is bounded by

\sum_{t=1}^n \mathbb E[\Delta_{A_t}\mathbf 1_{\neg G_t}] \le \Delta_{\max}\sum_{t=1}^n \Pr(\neg G_t) \le \Delta_{\max}\sum_{t=1}^\infty O\!\left(\frac{1}{t^4}\right) = O(1),

so the contribution from the “bad” events $\neg G_t$ is a constant independent of $n$ .

On the good events $G_t$ , regret depends on how often suboptimal arms are pulled. Let $N_n(a_i)$ be the number of pulls of arm $a_i$ up to round $n$ . Expanding $\Delta_{A_t}$ over arms gives

\Delta_{A_t} = \sum_{i=2}^k \Delta_i\,\mathbf 1\{A_t = a_i\},

R_{\operatorname{UCB}}(n) = \sum_{t=1}^n \mathbb E[\Delta_{A_t}\mathbf 1_{G_t}] + O(1) \\ = \sum_{t=1}^n \sum_{i=2}^k \Delta_i\,\mathbb E\bigl[\mathbf 1\{A_t = a_i\}\mathbf 1_{G_t}\bigr] + O(1) \\ = \sum_{i=2}^k \Delta_i\,\mathbb E\Bigl[\sum_{t=1}^n \mathbf 1\{A_t = a_i\}\mathbf 1_{G_t}\Bigr] + O(1) \\ \le \sum_{i=2}^k \Delta_i\,\mathbb E\Bigl[\sum_{t=1}^n \mathbf 1\{A_t = a_i\}\Bigr] + O(1) = \sum_{i=2}^k \Delta_i\,\mathbb E\bigl[N_n(a_i)\bigr] + O(1).

Whenever a suboptimal arm $a_i$ is chosen at round $t$ and $G_t$ holds, the selection rule implies

\hat\mu_{a_i} + B_t(a_i) \ge \hat\mu_{a_1} + B_t(a_1),

which together with $\bigl|\hat\mu_{a_i} - \mu_i\bigr|\le B_t(a_i)$ and $\bigl|\hat\mu_{a_1} - \mu_1\bigr|\le B_t(a_1)$ yields

\mu_i + 2B_t(a_i) \ge \mu_1 \quad\Longrightarrow\quad B_t(a_i) \ge \frac{\Delta_i}{2}.

Recalling that with $\alpha=2$ we have $B_t(a_i) = \sqrt{\tfrac{2\log t}{N_t(a_i)}}$ , this implies

\sqrt{\frac{2\log t}{N_t(a_i)}} \ge \frac{\Delta_i}{2} \quad\Longrightarrow\quad N_t(a_i) \le \frac{8\log t}{\Delta_i^2},

and hence in particular

N_n(a_i) \le \frac{8\log n}{\Delta_i^2}.

Plugging this back into the regret expression and combining with the constant contribution from bad events, we obtain

R_{\operatorname{UCB}}(n) = O\!\left(\sum_{i=2}^k \frac{\log n}{\Delta_i}\right).

In particular, if we let $\Delta = \min_{i\ge2}\Delta_i$ and denote the UCB regret at horizon $n$ by $R_{\mathrm{UCB}}(n)$ , then

R_{\mathrm{UCB}}(n) = O\!\left(\frac{\log n}{\Delta}\right).

Putting this together with the ETC analysis above, both $R_{\mathrm{ETC}}(n)$ and $R_{\mathrm{UCB}}(n)$ achieve a gap‑dependent logarithmic rate $O((\log n)/\Delta)$ , but ETC needs a well‑tuned exploration length $m$ (which in turn depends on the unknown gap), whereas UCB attains the same order of regret without any prior knowledge of $\Delta$ by adapting its exploration bonus online.

Many expositions (e.g., Introduction to Multi-Armed Bandits, pp. 5–6) present regret directly under expectation and sometimes blur the distinction between pathwise and expected quantities. The split into realized, random, and expected pseudo‑regret avoids that ambiguity. ↩︎
This section mainly refers to the ITCS course taught by Prof. Zhengfeng Ji. ↩︎

RL Note 1: Basics

Tue, 28 Oct 2025 15:23:52 +0800

Prologue

It’s been a while since I last updated this blog, so I’m kicking off a new series.

I’m currently taking a Reinforcement Learning (RL) course taught by Prof. Hongning Wang. I’m really enjoying his lectures and carefully prepared notes, and I’ve quickly grown fond of RL.

I’ve decided to turn the course material into a series of blog posts—not only to deepen my own understanding, but also in the hope that it helps other readers. The series will largely follow the Fall 2025 Reinforcement Learning course, but rather than simply repeating the lecture slides, I’ll add my own insights. I’ll also place extra emphasis on the mathematics—equations and proofs—which I find both challenging and especially interesting.

The major topics of this series are:

Basics of Reinforcement Learning
Multi-Armed Bandits
Markov Decision Processes
Dynamic Programming
Monte Carlo Methods
Temporal-Difference Learning
Policy Gradient Methods
Function Approximation
Deep Reinforcement Learning

I’ll cover each of these in its own post.

Today’s topic introduces the basic concepts of RL, which are foundational for everything that follows. As Prof. Wang noted in class, these fundamentals capture much of what RL is about.

Let’s begin our journey into Reinforcement Learning!

What is Reinforcement Learning?

In short, reinforcement learning is about an agent that continually updates its policy—how it selects actions—based on feedback from the environment, with the goal of maximizing future cumulative reward.

The figure below illustrates this process:

You now have a broad picture of RL. To go deeper, we need to clarify a few core concepts that will ground the discussions to come:

Action vs. Reward
State vs. Value
Policy
Model

Action vs. Reward

Definitions

These concepts are easy to understand, but they are two of the most important components in RL. First, the definitions:

Definition: Action

A choice made from the options presented by the environment.

Definition: Reward

The scalar feedback signal for the action taken.

These definitions seem simple, but a few points are worth noting.

What does ‘presented options’ mean?

This means the agent does not decide the options: they are given by the environment, and the agent can only choose among them.

The Goal of RL

Nowadays, when researchers train LLMs with RL methods, they often use reward as a key metric to evaluate performance, which leads to a common misconception: that RL tries to maximize the reward of a single action. This is wrong because the true feedback may be delayed. Some actions that seem good now may lead to increasingly worse situations, while some actions that seem not so good now may turn out to be wise after a few steps. Hence, as summarized before:

The Goal of Learning From the Perspective of Reward

The goal of learning is to maximize cumulative reward.

In other words, all goals can be described by the maximization of expected cumulative reward. However, because reward is often manually designed, this statement assumes the reward design is good enough. Reward really counts!

State vs. Value

These two concepts are closely related to Action vs. Reward. First, let’s see how an agent takes an action in RL:

How does the agent take an action?

As shown in the figure above, we can abstract the RL process as:

o_1, a_1, r_1, \ldots, o_t, a_t, r_t, \ldots, o_T, a_T, r_T

where $o$ denotes observations, $a$ denotes actions, and $r$ denotes rewards; $t$ is the current time step; $k < t$ refers to the history, and $k > t$ refers to the predicted future.

This sequence describes the standard RL loop: the agent observes the environment, chooses an action, receives feedback, and then observes the updated environment, and so on so forth.

This sequence highlights two key ideas:

We care about the history, because it helps indicate which actions are good or bad.
We care about the future, because we want the agent to perform well over time, not just at the current step.

This leads to the following definitions:

Definitions

Definition: State

A function of the history.

s_t = f(o_0, a_0, r_1, o_1, a_1, r_2 \ldots, o_t)

At time $t$ , we receive an observation $o_t$ , decide to take an action $a_t$ , and then receive a reward $r_{t+1}$ and a new observation $o_{t+1}$

Definition: Value

A function that evaluates how good the current state is for the future. We usually consider two value functions:

State–action value:
$Q_\pi(s_t, a_t) = \mathbb{E}_\pi\left[ \sum_{k=t+1}^{T} \gamma^{k-t-1} R_k\mid S_t=s_t,A_t=a_t \right]$
where $R_k$ represents the random varaible for reward.
State value:
$v_\pi(s_t) = \mathbb{E}_{a_t \sim \pi(\cdot\mid s_t)}\left[ Q_\pi(s_t, a_t) \right]$

Further notes:

What does the subscript $\pi$ mean?

It denotes the policy, i.e., the probability of taking each action in the current situation(we will give the formal definition later). In the definitions above, the policy $\pi$ is fixed, so we evaluate value under a given policy. We may later adjust the policy based on the value function.

How to interpret the $\mathbb{E}$ ?

For the state–action value: environments are complex; future rewards and observations are uncertain given a current state–action pair. We therefore take the expectation over all possible future trajectories.
For the state value: the expectation is over actions (according to $\pi$ ) to evaluate how good the current state is.

What does $\gamma$ mean?

It is the discount factor for future rewards. Because future outcomes are less certain than immediate ones, we gradually downweight them. Typically, $0 \leq \gamma < 1$ .

Hence, the goal of RL can be clarified more accurately as:

The Goal of Learning From the Perspective of Value

The goal of learning is to find the states with highest value.

Policy

We introduced the idea of policy earlier; here is the precise definition.

Definition: Policy

A policy maps each state to a probability distribution over actions:

\pi : \mathcal S \to \Delta(\mathcal A), \quad \text{so } \pi(a \mid s) = \Pr(A=a \mid S=s).

Here $\Delta(\mathcal A)$ denotes the probability simplex over $\mathcal A$ —all non‑negative vectors on $\mathcal A$ whose entries sum to 1.

Model

Definition: Model

The model is the agent’s estimated view of the environment.

Model vs. Environment

The environment exists objectively, whereas the model is merely the agent’s estimate, inferred from history.

Conclusion

We have sketched the core ingredients of reinforcement learning—reward, value, policy, and model—and highlighted why cumulative return matters more than any single outcome.

Takeaways

Actions are choices from environment-provided options, and rewards are the scalar feedback signals we must consider cumulatively, not step by step.
States summarize history, while value functions estimate future return.
A policy maps each state to a distribution over actions, defining the agent’s behaviour.
The model is the agent’s estimate of the environment.

For a worked example that ties these ideas together, I recommend pages 8–12 of Prof. Wang’s Lecture 1 slides; they walk through Dijkstra’s algorithm from an RL perspective with carefully prepared animations—far better than screenshots I could share (and, honestly, I’m too lazy to recreate them here).

Thanks for reading, and I hope this series continues to help with your own learning journey.

Chezmoi——一种优雅的点文件管理工具

Wed, 30 Apr 2025 01:43:02 +0800

想必经常在多台开发机器之间切换的朋友都有这样的烦恼：当使用一台新的服务器或者起了一份新的 Docker 时，从 0 开始的开发环境让人极不习惯，重新配一份又过于浪费时间。笔者经过一番检索，发现了 Chezmoi 这个优雅的点文件管理工具，几乎完美解决了上述痛点，因此在这里安利给大家。

概述

Chez Moi 在法语中是“在我家”的意思。顾名思义，该工具也是用来管理位于家目录下各种配置文件（主要也就是各种点文件）的，并集成了诸如多设备同步、加密和脚本等各种高级功能。

chezmoi 的哲学是将所有文件托管到~/.local/share/chezmoi这个 Git 仓库，我们可以将已有的文件添加其中，也可以将其中的文件应用到实际的家目录，或者托管到 GitHub 等平台上。

安装

最简洁的安装方法只需要运行如下一行命令：

1

sh -c "$(curl -fsLS get.chezmoi.io)"

当然，身为大陆用户，有的时候连接 GitHub 有困难，可以运行如下命令：

1

sh -c "$(curl -fsLS https://github.moeyy.xyz/https://raw.githubusercontent.com/Daucloud/dotfiles/refs/heads/main/install.sh)"

如果你已经在https://github.com/$GITHUB_USERNAME/dotfiles托管了你的点文件，也可以运行如下命令一键安装+配置：

1

sh -c "$(curl -fsLS get.chezmoi.io)" -- init --apply $GITHUB_USERNAME

基本使用

如果你是第一次使用 chezmoi，可以遵循如下步骤开始：

chezmoi init: 该命令会在~/.local/share/chezmoi下初始化一个 Git 仓库。
chezmoi add ~/.bashrc: 将~/.bashrc纳入 chezmoi 的管理。其在~/.local/share/chezmoi下对应的文件是dot_bashrc, 下文统一称为源文件。
chezmoi edit ~/.bashrc: 编辑源文件。
chezmoi diff: 查看源文件和实际家目录下的对应文件的差异。
chezmoi -v apply: 将源文件应用到家目录。其中-v选项会显示对实际文件所做的更改，建议添加。
chezmoi cd: 进入~/.local/share/chezmoi.
git add .然后git commit: 提交更改。
在 GitHub 下新建一个仓库(建议命令为dotfiles，这样在新机器的安装的时候可以使用sh -c "$(curl -fsLS get.chezmoi.io)" -- init --apply $GITHUB_USERNAME一键应用配置)，然后:

1
2
3


git remote add origin git@github.com:$GITHUB_USERNAME/dotfiles.git
git branch -M main
git push -u origin main

至此，你已经完成了对点文件的托管！

在新机器上应用配置

chezmoi init git@github.com:$GITHUB_USERNAME/dotfiles.git
chezmoi merge $FILE: 将当前机器的$FILE合并到源文件
chezmoi apply -v.
chezmoi update -v: 如果git@github.com:$GITHUB_USERNAME/dotfiles.git有新的提交，可用该命令更新源文件。

模版

Chezmoi 使用 go 语言编写，因此可以使用 go 模版语法。可以使用如下命令将文件设为模版：

1

chezmoi add ~/.gitconfig --template

这会在~/.local/share/chezmoi下创建一个名为dot_gitconfig.tmpl的文件。模版有很多妙用，譬如我在机器 A 上的 Git 使用邮箱A@test.com，在机器 B 上使用B@test.com，则我们可以这样实现：

chezmoi edit ~/.gitconfig，并编辑其中内容如下：

1
2


[user]
 email = {{ .email | quote }}

chezmoi cd
vim .chezmoi.toml.tmpl，编辑其中内容如下：

1
2
3
4


{{ $email := promptStringOnce . "email" "What is your email address" -}}

data:
 email: {{ $email | quote }}

这样，在 A 上chezmoi apply并询问 What is your email address 时，我们键入 A@test.com，在B中键入 B@test.com即可。

可以使用 chezmoi data 查看各种内置变量。除此之外，模版还有很多妙用，其余可参看 chezmoi 官方文档。

脚本

chezmoi 支持添加run_开头的脚本，用于在每次 chezmoi apply 时执行。其中按照文件名开头的不同分为如下三种类型:

run_开头：每次chezmoi apply都会运行
run_onchange_开头：如果文件内容相比上次有所变化，则chezmoi apply时会运行
run_once_开头：如果该内容的文件从未执行过，则chezmoi apply时会运行

下面举个例子来说明脚本的使用：身为大陆用户，难以避免需要给pip和conda等配置镜像源，Oh-My-Tuna 就是一个方便的选择。我们可以写一个如下的脚本，让每次 chezmoi 自动为我们配置好镜像源：

1

wget https://tuna.moe/oh-my-tuna/oh-my-tuna.py -O ~/.local/share/chezmoi/run_once_configure_tuna_mirrors.py

可将所有脚本在 ~/.local/share/chezmoi/.chezmoiscripts下统一管理，保持目录清洁。

加密

chezmoi 支持对管理的文件进行加密。下面以 age 加密为例：

chezmoi cd
创建加密的私钥(可能需要先参照 age 官网安装 age 加密工具):

1
2
3
4


$ age-keygen | age --armor --passphrase > key.txt.age
Public key: age193wd0hfuhtjfsunlq3c83s8m93pde442dkcn7lmj3lspeekm9g7stwutrl
Enter passphrase (leave empty to autogenerate a secure one):
Confirm passphrase:

注意记录公钥(age193wd0hfuhtjfsunlq3c83s8m93pde442dkcn7lmj3lspeekm9g7stwutrl)，后面会用到。

echo key.txt.age >> .chezmoiignore: 防止 chezmoi 在家目录下创建 key.txt.age
创建一个脚本，使得第一次 chezmoi apply 时解密 key.txt.age为私钥 key.txt

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


$ chezmoi cd
$ cat > run_once_before_decrypt-private-key.sh.tmpl <<EOF
#!/bin/sh

if [ ! -f "${HOME}/.config/chezmoi/key.txt" ]; then
 mkdir -p "${HOME}/.config/chezmoi"
 chezmoi age decrypt --output "${HOME}/.config/chezmoi/key.txt" --passphrase "{{ .chezmoi.sourceDir }}/key.txt.age"
 chmod 600 "${HOME}/.config/chezmoi/key.txt"
fi
EOF

创建一份模版配置：

1
2
3
4
5
6


$ cat > .chezmoi.toml.tmpl << EOF
encryption = "age"
[age]
 identity = "~/.config/chezmoi/key.txt"
 recipient = "age193wd0hfuhtjfsunlq3c83s8m93pde442dkcn7lmj3lspeekm9g7stwutrl"
EOF

.chezmoi.<format>.tmpl是 chezmoi 的一类特殊文件，用于创建配置文件 ~/.local/share/chezmoi/chezmoi.<format>

添加想要加密的文件

1

chezmoi add ~/.ssh/id_rsa --encrypt

如此一来，便不再需要在新的机器上配置各种 ssh 公私钥对了！

杂项

对于 Oh-My-Zsh 用户，由于 Oh-My-Zsh 的配置都在 ~/.oh-my-zsh中管理，所以可以运行 chezmoi add ~/.oh-my-zsh --exact --recursive将整个目录纳入 chezmoi 管理。

其实 chezmoi 官方提供了每次动态从 Oh-My-Zsh 官方下载的方法¹，但我觉得不如上面的方法简洁。

欢迎参考我的配置： https://github.com/Daucloud/dotfiles

参见 https://www.chezmoi.io/user-guide/include-files-from-elsewhere/ ↩︎

使用Latex Worksop时，如何引入Minted宏包？

Tue, 15 Apr 2025 21:49:15 +0800

我习惯使用VScode(现在是Cursor)+Latex Workshop工具链来书写Latex，但是最近引入Minted宏包时遇到了一些困难，并困扰了我一阵(详见The latexmk recipe can find pygmentize, but the xelatex and pdflatex recipes cannot.)，因此打算记录一下。

pip install Pygments. Minted依赖pygmentize来提供语法高亮支持。
打开VScode的settings.json，找到（或添加）latex-workshop.latex.tools配置，并在各个tools的args选项中添加-shell-escape选项。兹列出我的配置如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70


{
 "latex-workshop.latex.tools": [
 {
 "name": "xelatex",
 "command": "/Library/TeX/texbin/xelatex",
 "args": [
 "-shell-escape",
 "-synctex=1",
 "-interaction=nonstopmode",
 "-file-line-error",
 "%DOCFILE%"
 ]
 },
 {
 "name": "pdflatex",
 "command": "/Library/TeX/texbin/pdflatex",
 "args": [
 "-synctex=1",
 "-interaction=nonstopmode",
 "-file-line-error",
 "-shell-escape",
 "%DOCFILE%"
 ]
 },
 {
 "name": "latexmk",
 "command": "latexmk",
 "args": [
 "-synctex=1",
 "-interaction=nonstopmode",
 "-file-line-error",
 "-pdf",
 "-xelatex=xelatex -shell-escape %O %S",
 "-outdir=%OUTDIR%",
 "%DOC%"
 ]
 },
 {
 "name": "bibtex",
 "command": "/Library/TeX/texbin/bibtex",
 "args": ["%DOCFILE%"]
 }
 ],
 "latex-workshop.latex.recipes": [
 {
 "name": "XeLaTeX",
 "tools": ["xelatex"]
 },
 {
 "name": "PDFLaTeX",
 "tools": ["pdflatex"]
 },
 {
 "name": "BibTeX",
 "tools": ["bibtex"]
 },
 {
 "name": "LaTeXmk",
 "tools": ["latexmk"]
 },
 {
 "name": "xelatex -> bibtex -> xelatex*2",
 "tools": ["xelatex", "bibtex", "xelatex", "xelatex"]
 },
 {
 "name": "pdflatex -> bibtex -> pdflatex*2",
 "tools": ["pdflatex", "bibtex", "pdflatex", "pdflatex"]
 }
 ]
}

务必确保--shell-escape选项在%DOCFILE%或%DOC%之前！

Hello World

Sun, 23 Mar 2025 19:36:09 +0800

搭好了新博客，从 Hexo 转 Hugo 了！

希望这次能够坚持写作:(

高代选讲期末重点整理

Tue, 18 Jun 2024 00:00:00 +0800

此文为宗正宇老师在最后一节课所划六个重点的整理，权作期末预习(x)

求特征多项式和极小多项式

特征多项式

$f(\lambda)=\left|\lambda I-A\right|$

极小多项式

方法 1

定理 7.4

极小多项式 $m(\lambda)$ 和特征多项式 $f(\lambda)$ 含有完全相同的根集和不可约因子集。

一般来说考试涉及的矩阵并不复杂，维数也较小，因此可以根据定理 7.4，从小到大尝试 $f(\lambda)$ 的因式 $a(\lambda)$ 是否满足 $a(A)=O$ 以确定最小多项式

方法 2

求出矩阵 $A$ 的 Jordan 标准型，则 $(\lambda-\lambda_i)$ 项在 $m(\lambda)$ 中的重数即为特征值为 $\lambda_i$ 的 Jordan 块的最大阶数；对于广义 Jordan 块，可以直接取为相应的不可约因式

计算复方阵的 Jordan 标准型及可逆矩阵 P

Jordan 标准型计算

方法 1

求出特征多项式 $f(\lambda)$
则 $\lambda_i$ 的个数为 $r_i=n-rank(A-\lambda_iI)$ ，也即 $f(\lambda)$ 中 $(\lambda-\lambda_i)$ 的次数
阶数为 $t$ 的 Jordan 块的个数为 $r_i(t)=rank(A-\lambda_iI)^{t+1}+rank(A-\lambda_iI)^{t-1}-2rank(A-\lambda_iI)^t$

解释：注意到 $rank(A-\lambda_iI)^m$ 中仅含有原本阶数大于 $m$ 的 $\lambda_i$ 的 Jordan 块，故 $rank(A-\lambda_iI)^m-rank(A-\lambda_iI)^{m+1}$ 恰好为阶数大于 $m$ 的 $\lambda_i$ 的 Jordan 块的个数，因此 $\left(rank(A-\lambda_iI)^{m-1}-rank(A-\lambda_iI)^m\right)-\left(rank(A-\lambda_iI)^m-rank(A-\lambda_iI)^{m+1}\right)$ 为阶数大于 $m-1$ 的 $\lambda_i$ 的 Jordan 块的个数和阶数大于 $m$ 的 Jordan 块的个数之差，换言之即为阶数为 $m$ 的 Jordan 块的个数

一些 tricks:
1. 由于考试涉及的 Jordan 块通常维数不大，而 $m(\lambda_i)$ 则确定了各 $\lambda_i$ 的 Jordan 块的最大阶数，通过此往往就可求出 Jordan 标准型
2. 可以通过计算 $dim Ker(\lambda_iI-A)$ 来获知 $\lambda_i$ 的 Jordan 块的个数（ $Ker(\lambda_iI-A)$ 中的每个向量都可以作为一条 Jordan 链的终结）

方法 2

求出 $A$ 的初等因子组，则 Jordan 标准型为

J=\begin{bmatrix}J\left(p_1^{k_1}\right)&&\\&\ddots&\\&&J\left(p_s^{k_s}\right)\end{bmatrix}

注：该方阵称为第三种相似标准型，除此之外，还有第一类相似标准型（有理标准型）和第二类相似标准型（初等因子友阵型）:

C=\begin{bmatrix}C\left(d_1\right)&&\\&\ddots&\\&&C\left(d_r\right)\end{bmatrix}\\G=\begin{bmatrix}C\left(p_1^{k_1}\right)&&\\&\ddots&\\&&C\left(p_s^{k_s}\right)\end{bmatrix}

可逆矩阵 P 的求取

方法 1

求出 $Ker(A-\lambda_iI)$ ，其中的向量可以作为 $\lambda_i$ 的 Jordan 链的终结
但是需要注意，并不是每个向量都可以形成阶数正确的 Jordan 块。譬如，设 $Ker\left(A-\lambda_iI\right)$ 由 $\alpha_1$ 和 $\alpha_2$ 张成。如果一个 Jordan 块为二阶，那么我们需要设此 Jordan 链的终结为 $s\alpha_1+t\alpha_2$ ，并确保 $\left(A-\lambda_iI\right)x=s\alpha_1+t\alpha_2$ 有解（通过考虑增广矩阵 $\begin{bmatrix}A&s\alpha_1+t\alpha_2\end{bmatrix}$ ）

举一道例题，便于理解

问：求此复数域上方阵的 Jordan 标准型和可逆矩阵 P：

\begin{bmatrix}1&-3& 0&3\\-2&-6& 0&13\\0&-3& 1&3\\-1&-4& 0&8\\\end{bmatrix}

解：

有

f(\lambda)=(\lambda-1)^4,\quad m(\lambda)=(\lambda-1)^3

故 Jordan 标准型为

\begin{bmatrix} 1&0&0&0\\ 1&1&0&0\\ 0&1&1&0\\ 0&0&0&1 \end{bmatrix}

并且

Ker(A-I)=span\left( \begin{bmatrix}3\\1\\0\\1\end{bmatrix}, \begin{bmatrix}0\\0\\1\\0\end{bmatrix} \right)

取

\alpha_4=\begin{bmatrix}3\\1\\0\\1\end{bmatrix},\quad \alpha_3=\begin{bmatrix}3s\\s\\t\\s\end{bmatrix}

欲使 $(A-I)\alpha_2=\alpha_3$ 有解，考虑增广矩阵：

\begin{bmatrix} 0&-3&0&3&3s\\ -2&-7&0&13&s\\ 0&-3&0&3&t\\ -1&-4&0&7&s \end{bmatrix}

可知 $3s=t$ 。不妨取 $s=1$ ，故

\alpha_3=\begin{bmatrix}3\\1\\3\\1\end{bmatrix},\quad \alpha_2=\begin{bmatrix}3u+3\\v-1\\0\\0\end{bmatrix}

再令 $(A-I)\alpha_1=\alpha_2$ 有解，考虑增广矩阵：

\begin{bmatrix} 0&-3&0&3&3u+3\\ -2&-7&0&13&u-1\\ 0&-3&0&3&v\\ -1&-4&0&7&w \end{bmatrix}

有 $3u+3=v,\ 2w-\frac{v}{3}=u-1$ 。不妨取 $u=1,v=6,w=1$ ，故

\alpha_2=\begin{bmatrix}6\\0\\6\\1\end{bmatrix},\quad \alpha_1=\begin{bmatrix}7\\-2\\0\\0\end{bmatrix}

故知

P=\begin{bmatrix} 7&6&3&3\\ -2&0&1&1\\ 0&6&3&0\\ 0&1&1&1 \end{bmatrix}

方法 2

由于 $\lambda I-A$ 和 $\lambda I-J$ 有相同的 Smith 标准型，而 Smith 标准型有固定的算法求解，因此可以求 $P_1^{-1}\left(\lambda I-A\right)P_1=P_2^{-1}\left(\lambda I-J\right)P_2=S$ ，则 $P=P_2P_1^{-1}$

lambda-矩阵对角化；初等因子、不变因子和 Jordan 标准型的关系

lambda-矩阵对角化

课本有详尽的算法。简而言之即是不断降低 $a_{11}$ 的次数

初等因子、不变因子与 Jordan 标准型的关系

三种基本因子的概念

对于 $\lambda-$ 矩阵

不变因子：Smith 标准型对角线上的所有非零元素： $d_1(\lambda),…,d_r(\lambda)$
行列式因子：所有 k 阶非零子行列式的首 1 最大公因子： $D_1,…,D_r$
初等因子：不变因子的准素因子全体（1 不计入）

不变因子和行列式因子互相决定： $D_k=\prod_{i=1}^k d_i,\ d_k=\frac{D_k}{D_{k-1}}$

对于数字矩阵，其三种基本因子指的是 $\lambda I-A$ 的三种基本因子（但是 1 均不计入）

从初等因子求 Jordan 标准型

J=\begin{bmatrix}J\left(p_1^{k_1}\right)&&\\&\ddots&\\&&J\left(p_s^{k_s}\right)\end{bmatrix}

Gram-Schmidt 正交化

设 $A=\begin{bmatrix}v_1&v_2&\cdots&v_n\end{bmatrix}$
则其正交化结果 $Q=\begin{bmatrix}w_1&w_2&\cdots&w_n\end{bmatrix}$
其中：

w_1=v_1,\qquad w_k=v_k-\sum_{j=1}^{k-1}\frac{\langle w_j,v_k\rangle}{\langle w_j,w_j\rangle}w_j, \quad k=2,\dots,n

上学期的内容，不过多赘述

规范方阵谱分解

求出规范方阵 $A$ 的特征多项式，并解出特征值则 $U^{-1}AU=diag(\lambda_{1},\cdots,\lambda_n)$
求出 $Ker(\lambda_iI-A)$ 的一组基，对其施行 Gram-Schimidt 正交化
将上一步得到的所有向量排列起来，即得到酉方阵 $U$

Jordan 标准型的应用和可对角化条件

设 $A$ 为 n 阶复方阵，则如下命题等价：

$A$ 在 $\mathbb{C}$ 可对角化
$A$ 的几何重数为 n（有 n 个线性无关的特征向量）
$A$ 在复数域上的初等因子均为 1 次（即初等因子均无重根）
$A$ 的不变因子均无重根
$A$ 的极小多项式无重根
$\forall c\in\mathbb{C},rank(cI-A)=rank\left((cI-A)^2\right)$

此外，以上命题的一个充分条件是 $f(\lambda)$ 无重根

Chapter2 Using 🤗 Transformers

Tue, 14 May 2024 00:00:00 +0800

Behind the pipeline

pipeline() groups preprocessing, model inference, and postprocessing together.

Preprocessing with a tokenizer

A tokenizer converts raw text into vectors by:

splitting the input into tokens, such as words, subwords, and punctuation;
mapping each token to an integer;
adding any extra inputs the model needs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


from transformers import AutoTokenizer

checkpoint="checkoutName"
tokenizer=AutoTokenizer.from_pretrained(checkpoint)# all the preprocessing needs to be done exactly the same way as when the model was pretrained
raw_input="context"
inputs=tokenizer(raw_inputs,padding=True,truncation=True,return_tensors="pt")

# as for the return_tensors, pt: Pytorch, tf: TensorFlow, np: Numpy, jax: JAX
print(inputs)

### the results, is a dictionary, which contains two key-value pair
{
 "input_ids": tensor([1,1,1]), # the unique identifiers for each token
 "attention_mask": tensor([2,2,2])
}

Going through the model

The model converts input IDs into logits.

🤗 provides the AutoModel class, which corresponds to the hidden-states step.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


from transformers import AutoModel
checkout="checkoutname"
model=AutoModel.from_pretrained(checkout)
outputs=(**inputs)

# the Automodel converts the inputs(seen last part) into a three-dimensional vector:
"""
1. batch size: the number of sequences processed at a time
2. sequence length
3. Hidden size: usually very large(768, 3072, even more)
"""
print(outputs.last_hidden_state.size) # torch.Size([2,16,768])

There are also task-specific architectures. You can understand them as AutoModel followed by a task head:

ForCausalLM
ForMaskedLM
ForMultipleChoice
ForQuestionAnswering
ForSequenceClassification
ForTokenClassification
other 🤗 task heads

1
2
3
4
5


from transformers import AutoModelForSequenceClassification
checkout="checkoutName"
model=AutoModelForSequenceClassification.from_pretrained(checkout)
outputs=(**inputs)
print(outputs.logits.shape) #torch.Size([2,2]), the size is much smaller than the results of AutoModel

Postprocessing the output

Postprocessing converts logits into human-readable predictions, usually by turning raw unnormalized scores into probabilities.

Usually we use a softmax layer for this step.

1
2
3
4
5
6
7
8
9


import torch
predictions=torch.nn.functional.softmax(outputs.logits, dim=-1)
print(predictions)
'''
tensor([[5.0e-2,9.5e-1],[1.0e-1,9.0-1]],grad_fn=<SoftmaxBackward>)
'''

# we can inspect the id2label the following way:
model.config.id2label # {0:'NEGATIVE',1:'POSITIVE'}

Models

AutoClass and its relatives are wrappers that can infer the architecture from the checkpoint.

Creating A Transformer

1
2
3
4


from transformers import BertConfig, BertModel,
config=BertConfig()
model=BertModel(config)# in this way, you will get a randomly initialized bert model, which will output gibberish
model=BertModel.from_pretrained("bert-base-cased")# the model is instantiate with the checkpoint trained by the bert team, which is less time-consuming and more environment-friendly; you can also replace the BertModel with AutoClass. Actually, it is more suggested to use AutoModel rather than a specific model, which will also work even if the architechture is different

Once you use the checkpoint, the weights are downloaded and cached to ~/.cache/huggingface/transformers.

You can use save_pretrained to save the model to disk:

1
2
3
4
5
6
7


model.save_pretrained("path/to/directory")
!ls path/to/directory
"""
config.json pytorch_model.bin

# the two files go hand in hand, the `config.json` contains the attributes necessary to build the architechture, and the `pytorch_model.bin` contains the weights(checkpoints) which are the parameters of your model
"""

Tokenizers

Algorithms for tokenization

Word-Based

Split words according to marks such as spaces and punctuation.
Map each word to an ID. The ID is determined through the vocabulary, which is usually very large. For example, an English vocabulary may be as large as 500,000.
Words outside the vocabulary are often represented by an unknown token such as [UNK] or <unk>. This loses information, so it is sensible to avoid unknown tokens as much as possible.

Character-Based

Split sentences by characters.

This method will definitely reduce the number of unknown tokens. However, it may also make sequences too long and the results less meaningful.

Subword Tokenization

Represent rare words as combinations of frequently used subwords.

This saves vocabulary space while preserving semantic meaning as much as possible, which is especially useful for agglutinative languages such as Turkish.

Examples:

Byte-level BPE: GPT-2
WordPiece: BERT
SentencePiece or Unigram: multilingual models

Loading and Saving

Loading and saving tokenizers is nearly the same as loading and saving models: use from_pretrained and save_pretrained.

Cached algorithms are similar to model architectures.
Cached vocabularies are similar to model weights.

Encoding

There are two steps when encoding text.

Tokenization

1
2
3
4
5
6


from transformers import AutoTokenizer

tokenizer=AutoTokenizer.from_pretrained("modelName")
tokens=tokenizer.tokenize("sequence")

print(tokens) # print the results of spilting

From tokens to input IDs

Models only accept tensors as inputs, so tokens must be mapped into numbers. The vocabulary is the dictionary that maps tokens to IDs.

Handling Multiple Sequences

Models expect a batch of inputs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model=AutoModelForSequenceClassification.from_pretrained("checkpointName")
tokenizer=AutoTokenizer.from_pretrained("checkpointName")

tokens=tokenizer.tokenize("raw inputs")
token_ids=tokenizer.convert_tokens_to_ids(tokens)
input_ids=torch.tensor(token_ids)

model(input_ids)
"""
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
"""
model([input_ids]) # this line will succeed

Padding the inputs

Batches must contain sequences of the same length because tensors are rectangular. This is why we introduce padding_id to pad inputs. You can access it with the tokenizer’s pad_token_id.

Attention masks

One key feature of Transformers is the attention layer that contextualizes each token. As a consequence, padding IDs can also affect the output, which is not expected.

Attention masks solve this. A value of 0 means the corresponding token should be ignored; 1 means it should be used.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


batched_ids = [
 [200, 200, 200],
 [200, 200, tokenizer.pad_token_id],
]

attention_mask = [
 [1, 1, 1],
 [1, 1, 0],
]

outputs = model(torch.tensor(batched_ids), attention_mask=torch.tensor(attention_mask))
print(outputs.logits)

Longer sequences

All Transformer models have a sequence length limit. If you want to pass a sequence longer than the limit, you should truncate your sequence or switch to a model that allows longer inputs.

Putting it all together

Put the tokenization steps together

As a review, there are three steps we need to tokenize raw inputs:

1

tokenizer.tokenize() -> tokenizer.convert_tokens_to_ids() -> torch.tensor()

For convenience, 🤗 transformers provides a high-level function that puts all these steps together: tokenizer() itself.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


from transformers import AutoTokenizer
tokenizer=AutoTokenizer.from_pretrained("checkpointName")

# 1
sequence="1"
input1=tokenizer(sequence) # valid

# 2
sequences=["1","2"]
input2=tokenizer(sequneces)

# 3 pad
input3=tokenizer(sequences, padding="longest") # pad to the maximum sequece length
input4=tokenizer(sequences, padding="max_length") # pad to the model maximum length
input5=tokenizer(sequences, padding="max_length", max_length=8) # pad to the specified maximum length

# 4 truncate
input6=tokenizer(sequences, truncation=True) # truncate the sequences longer than the model limit
input7=tokenizer(sequences, max_length=8, truncation=True) # truncate the sequences longer than the specified length

# 5 switch the type of tensors returned
input8=tokenizer(sequences, padding=True, return_tensors="pt") # pt stands for the Pytorch, tf for TensorFlow, np for NumPy

Special words

Some models add special tokens such as [CLS] at the beginning; some add [SEP] at the end; some add both; and some add none.

Wrapping up: From tokenizer to model

1
2
3
4
5
6
7
8
9


import torch
from transfomers import AutoTokenizer, AutoModel

tokenizer=Autokenizer.form_pretrained("checkpointName")
model=AutoModel.form_pretrained("checkpointName")

sequence="1"
Input=tokenizer(sequence)
Output=model(**Input)

Chapter1 Transformer Models

Fri, 10 May 2024 00:00:00 +0800

最近打算入门 NLP，在自学 🤗 的 NLP Course，但是感觉自己过于摆烂了。于是打算边学边做笔记，争取在期末之前把本课程学完

Pipeline

the most basic object in the 🤗 Transformers llibrary
It can:
- feature-extraction(get a vector representing the text)
- fill-task
- ner(entity recognition)
- question-answering
- sentiment-analysis
- summarization
- text-generation
- translation
- zero-shot classification

Zero-shot Classification

you can casually assign the labels

1
2
3
4
5


classifier = pipeline("zero-shot-classification")
classifier(
 "I play Genshin Impact!",
 candidate_labels=["op","2-dimensional"]
)

Text Generation

involves randomness

1
2
3
4
5
6


generator=pipeline("text-generation")
generator(
 "prompts",
 max_length=15, #the max length of the text
 num_return_sequence=3 #How many texts are gonna generated
)

Mask filling

1
2


unmasker=pipeline("fill-mask")
unmasker("I play <mask> Impact!",top_k=2) #top_k decides the time it does;<mask> depends on what model you are using

Named entity recognition

1

ner=pipeline("ner",grouped_entities=True)#'grouped_entities=True' is used to enable the model to put multi-words together

Question answering

answer a question using given context
extracting answers from the context instead of generating answers

1
2
3
4
5


question_answerer=pipeline("question-answering")
question_answerer(
 question="where do I work?",
 context="My name is Sylvain and I work at Hugging Face in Brooklyn"
)

Summarization

1
2
3
4
5
6


summarizer=pipeline("summarization")
summarizer(
 """
context
 """
)

Translation

1
2


translator=pipeline("translation",model="modelName")
translator("contex")

Brief intro to Transformers

General categorization

GPT-like: auto-regressive
BERT-like: auto-encoding
BART/T5-like: sequence-to-sequence

Transfer Learning

Pretraning

the act of traing a model from scratch

Fine-tuning

training on the top of pretrained models with a dataset specific to the target task

General architecture

Transformers model is generally composed of two blocks:
- Encoder: receives inputs and builds representations for them
- Decoder: use the outputs of encoder to generate target outputs
Various tasks requires different blocks:
- Encoder-only: sentence classfication; NER
- Decoder-only: text generation
- Encoder-decoder models or sequence-to-sequence models: translation or summarization

Attention layer

pay specific attention to certain words while ignoring others more or less

Original Architechture

the encoder translate all the words
the decoder is only allowed to translate by the past words; but later it can get all the outpus of encoer to better translate the word

漫步杂记

Sun, 17 Mar 2024 00:00:00 +0800

校园夜色图片来源：Mapio

喜欢在少人的夜，一个人漫无目的地在校园里走走停停。瘫在宿舍打了一下午的游戏，头脑昏昏沉沉。于是掷下手机，抄起外套，择宿舍楼西边的一条僻幽小径，开始在入夜的园子里随意游荡。正值冬春之交，北京的风已然收起了冬日硌人的棱角，佐以夜的些许清冷的寒意，迎面吹来，令人感到舒适又清醒。享受着风的感觉，心灵逐渐从日常生活的繁琐和世俗意义的压力中抽离，柔软的触角不知不觉向四周延伸开来：落叶随风移动时摩擦地面的簌簌声、校河反射路灯的粼粼波光、道旁宿舍楼中通亮的灯、黑夜中轻柔摇动的柳条、奇形怪状的光秃枝丫、含苞待放的粉色蓓蕾……心灵仿佛褪去了老成的外壳，对司空见惯的一切重又兴起了孩童般的好奇，目之所及、耳之所闻都是再新奇不过的事物。

肆意的外界体验唤醒了丰盈的内在感受，骨髓深处的孤独感一点点渗了出来，悄然漫上了心头。我是一个孤独的人，内心深处，始终有一部分是无法对人敞开、甚至我自己也不甚了然的。很多时候，我对这种孤独感是异常恐惧的：在某些难眠的深夜，我会被潮水般涌出的孤独窒塞住呼吸。我渴望有一名真正的灵魂知己能够分享一切、倾吐所有。可或是不敢，或是无缘，终究没能找到。但是，在这个随意漫步的晚上，我却逐渐学会了享受孤独。独自一人，可以随心所欲地停步和起步、随心所欲地蹲下来端详道旁的枯草、随心所欲地拾起脚边的枝条。原来孤独并不是什么可怕的事情。每个人生来孤独，这是一个人身为他自己而不是其他的什么人所必然带来的独一无二的体验。我当然仍渴望觅到一名知己，但我已学会拥抱这份孤独。

踱步到大礼堂北边的河道时，突然有一阵极猛烈的风顺着我前进的方向吹来。我立住片刻，感受着风在耳边的呼啸声和强风下紧贴身体的衣物。随后我转过身，张开双臂，闭目体会寒冷的气流冲击面颊时的微痛。我逆风走了几步，随后又转过身去，顺风走了起来。我越走越快，越走越跳跃，越走越轻盈。我忍不住笑了起来，一开始是微笑，后来是咧嘴笑，最后竟演变成了哈哈大笑。我一边笑，一边大声说：“我还活着！”后来风停了，我还在大声笑着。周围没什么人，见证者只有河水、柳树和礼堂。

回宿舍时，我走的是学堂路。学堂路人流不小，走在道路的边缘，时时有自行车从身旁驰过。我饶有兴致地看着各式各样的自行车远去的身影，听着偶尔传来的一两句玩笑和谈话声，发散的思绪重又回到了日常生活，只是多上了几分释然、坦荡和宁静。漫步的趣味，当真言之不尽。

可以判断素数的正则表达式

Thu, 01 Feb 2024 00:00:00 +0800

最近读到一个可以判断素数的正则表达式（匹配成功则不是素数）：

1

/^1?$|^(11+?)\1+$/

这么精炼，颇有些出乎我的意料。因为在我的印象中大多数正则表达式都十分丑陋，比如匹配邮箱的正则表达式：

1

(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])

然而，仔细阅读了一下原理后却发现它的确可以，不过需要一步预处理，即把十进制数字转换为“1的数组”：0为空字符串，1为1，2为11，3为111……
至此，就可以开始我们的匹配：

/^1?$/ 很好理解，匹配到的是空字符串（0）或 1，自然不是合数；
/^(11+?)\1+$/ 则正是该表达式的精妙所在。其利用到了正则表达式匹配时回溯的特性。让我们具体解释一下：

(11+?)匹配的是至少两个1，但是因为进行的是懒惰匹配，所以最开始匹配到的是11，并被捕获到了分组\1中。后面部分中，如果11重复了一次或者更多次，那么就是合数。这又是为什么呢？其实非常容易理解：如果数字 $a$ 匹配成功，意味着 $a$ 化作的“1的数组”恰好由不少于2个11组成，也就是， $\exists b>1\land b \in \mathbb N, a=2b$ ，这自然意味着 $a$ 是一个合数(在这种情况下， $a$ 还是偶数)。
如果11匹配失败了呢？这就是本法最核心的部分了：匹配器会进行回溯，对下一个满足(11+?)的111进行\1+匹配的尝试。同理上一步，如果匹配成功，该数字大于3且含有一个因数3，自然是合数。
接下来，“匹配 - 回溯”不断进行。如果 $n$ 是合数，会在匹配不超过 $\sqrt{n}$ 次后成功；反之，会一直匹配到第 $n$ 次，最后匹配失败。

可见，/^1?$|^(11+?)\1+$/的实现想法是非常朴素的：列举比 $n$ 小的所有正整数 $i$ ，判断 $n$ 是否可以被 $i$ 整除。比如：

1
2
3
4
5
6
7


//判断一个自然数是否为素数
bool isPrime(int n){
    if (n == 0 || n == 1)return false;
    if (n == 2)return true;
    for (int i = 2; i <= sqrt(n); i++)if (n % i == 0)return false;
    return true;
}

但是，得到这么精炼的表达式的关键实则在于对于正则表达式匹配机制的深刻理解(懒惰匹配和回溯法的结合)，而这正是值得我们深思和学习的地方。

生成对抗网络

Thu, 18 Jan 2024 00:00:00 +0800

~~本文是数据科学导论期末大作业展示的报告，拿来随便水一篇博客hh~~

0 引言

生成对抗网络 (Generative Adversarial Network, GAN) 是一种十分流行的机器学习模型。自2014年Ian Goodfellow等人首次提出以来，GAN迅速在学术界和工业界引发了热烈的反响，许多有影响力的工作层出不穷。图一是累积的GAN论文数量，其火热程度可见一斑。

Figure 1: Cumulative number of GAN papers

本文将对GAN做简要介绍，主要包括GAN原始模型、经典变体和应用成就。

1 原始模型

1.1 基本思想

GAN全称为生成对抗网络。顾名思义，其在本质上是一种生成模型，它最突出的特点是使用判别器与生成器进行对抗式训练，后者负责产生贴近真实的数据，前者则负责努力辨别生成数据和真实数据。当训练迭代至一定次数后，生成器产生的数据便足够逼真，训练目的因而得到实现。

对此，GAN的原始论文¹中给出了一个通俗的解释：

生成器是“假钞制造团伙”，负责制造能在市场上流通的假钞；判别器则是“警察”，负责识破假钞。假钞制造团伙和警察之间不断竞争，警察的鉴别能力和团伙的造假能力都不断提升，最终，造假团伙制造的假钞足以以假乱真：“假钞变成了真钞”。

GAN就是在制造足以以假乱真的假钞。

1.2 基本步骤

概括来说，GAN的训练主要分为以下三步：

固定生成器 $G$ ，训练判别器 $D$ ： 使 $D$ 尽可能准确地识别出样本究竟是由生成器生成的（来自 $p_z(z)$ ）还是来自真实数据集（来自 $p_{data}(x)$ ）
固定判别器 $D$ ，训练生成器 $G$ ： 使 $G$ 生成的样本( $G(z)$ )尽可能贴近真实样本，即让生成器 $G$ 骗过判别器 $D$
迭代： 循环1. 2.至一定次数，此时生成器 $G$ 产生的样本足以“以假乱真”，判别器 $D$ 辨认真实样本和生成样本成功的概率都为 $\frac1 2$ ，二者达到了纳什平衡²。

训练示意图：

Figure 2: 训练过程示意图

图中， $z$ 轴是输入生成器的随机噪声，其通过生成器(箭头)映射到样本集 $x$ 轴。黑色散点是真实数据集；绿色实线是生成样本集，蓝色虚线是判别器（越远离 $x$ 轴，其值越接近于1，即其认为该样本来自真实数据集的概率越大）。

(a)中，判别器曲线(蓝色虚线)较为颠簸，判别效果较差，因此(a)到(b)首先对判别器进行训练；(b)到(c)则对生成器进行训练，使得生成样本曲线(绿色实线)更贴近真实样本集(黑色散点)。当迭代此二步到一定次数后，绿色实线和黑色散点接近重合，判别器曲线也恒为 $1 \over 2$ ，即无法将生成样本和真实数据进行区分。

1.3 具体算法

记号说明

生成器和判别器均为多层神经网络： $G(z;\theta _g)$ 和 $D(x;\theta _d)$
$G$ 通过参数 $\theta_g$ 将噪声 $z$ 映射为生成样本 $G(z;\theta _g)$
$D$ 通过参数 $\theta_d$ 将样本映射为 $[0,1]$ 的标量 $D(x;\theta _d)$ ，其值越接近1表示为真实数据的概率越大
目标，求解值函数 $V(D,G)$ 的极小极大值，即：

\min_G\max_D V(D,G)=\mathbb{E}_{x\sim p_{data}(x)}[\log D(x)]+\mathbb{E}_{z\sim p_z(z)}[\log(1-D(G(z)))]

具体求解步骤

1
2
3
4
5
6
7
8
9


for 1 to n do
 for 1 to k do
 从噪声集取出 m 个样本 {z^(1), z^(2), ..., z^(m)}
 从真实数据集中取出 m 个样本 {x^(1), x^(2), ..., x^(m)}
 使用梯度上升法更新判别器的参数 theta_d
 end for
 从噪声集中取出 m 个样本 {z^(1), z^(2), ..., z^(m)}
 使用梯度下降法更新生成器的参数 theta_g
end for

更新判别器的参数 $\theta_d$ 时使用梯度上升法³：

\nabla_{\theta_d}\frac{1}{m}\sum_{i=1}^{m}[\log D(x^{(i)})+\log D(1-D(G(z^{(i)})))].

更新生成器的参数 $\theta_g$ 时使用梯度下降法：

\nabla_{\theta_d}\frac{1}{m}\sum_{i=1}^{m}[\log (1-D(G(z^{(i)})))].

k是一个超参数；之所以更新k次 $D$ 才更新1次 $G$ ，是因为只有判别器足够好，生成器的更新才准确有效。

2 经典变体

GAN作为风靡一时的机器学习模型，自然少不了各种改进和变体。本部分选取了三种较有代表性和影响力的变体进行介绍：CGAN、DCGAN和WGAN。

2.1 CGAN

原始 GAN 模型⁴中，输入生成器的是随机噪声，因此最后产生的图像的随机性也相对较大。倘若我们想要获得具有某种特征和标签的图像，就需要从生成的一大堆样本中进行挑选，较为繁琐。

CGAN(Conditional Generative Adversarial Networks)便是为了解决这一问题而产生的。其思想也十分朴素，即向生成器和判别器的输入中添加条件信息。这样一来，生成样本不仅需要足够逼真，还要满足特定的条件才能通过判别器：

\min_G\max_D V(D,G)=\mathbb{E}_{x\sim p_{data}(x)}[\log D(x|y)]+\mathbb{E}_{z\sim p_z(z)}[\log(1-D(G(z|y)))]

Figure 3: CGAN和GAN的对比

2.2 DCGAN

DCGAN⁵ (Deep Convolutional Generative Adversarial Networks) 全称为深度卷积对抗生成网络。顾名思义，其主要想法是将卷积神经网络(CNN)和GAN进行结合，在不改变GAN的基本原理的情况下较为有效地改善了其训练不稳定的问题。

DCGAN做出的主要改变有：

使用卷积层和转置卷积层：引入了转置卷积层和卷积层分别作为生成器和判别器网络的主要组件。
去除全连接层：用全卷积层代替。
批归一化(Batch Normalization)：生成器和判别器都使用BN层。
修改激活函数：生成器输出层使用Tanh，其余层使用ReLU；判别器均使用LeakyReLU

Figure 4: 生成器转置卷积层示意图

2.3 WGAN

WGAN⁶ (Wasserstein Generative Adversarial Networks) 引入了Wasserstein距离代替原来的JS散度作为GAN的损失函数，彻底解决了GAN训练不稳定的问题，是GAN发展史上里程碑式的工作之一。

Figure 5: WGAN算法

可见，WGAN做出的最主要改动，即是对损失函数的更换。此外，其还做出了如下改变：

判别器最后一层去掉sigmoid
每次更新判别器的参数之后把它们的绝对值截断到不超过一个固定常数c
不要用基于动量的优化算法（包括momentum和Adam），推荐RMSProp，SGD也行

改进虽然简单，但是成效巨大。

3 应用举例

3.1 「Edmond de Belamy」

在2018年末的佳士得纽约拍卖场上，来自巴黎的艺术团队Obvious使用GAN生成的画作「Edmond de Belamy」以超出估价40倍的43.25万美元成交，其右下角便印有纵横GAN领域的著名公式： $\min_G\max_D \mathbb{E}_{x}[\log D(x)]+\mathbb{E}_{z}[\log(1-D(G(z)))]$

1968年，毕加索曾说：”计算机是没有用的。它们只会告诉你答案”。但在同一场拍卖会上，没有一幅毕加索的画作成交价格超过了「Edmond de Belamy」，这不禁令人唏嘘不已。

Figure 6: 「Edmond de Belamy」

3.2 This Person Does Not Exist

thispersondoesnotexist.com 每次进入该网址，都会生成一张世界上并不存在的人脸，而这正是使用GAN进行生成的。

Figure 7: This Person Does Not Exist

3.3 二次元头像生成

使用GAN⁷，你可以随心所欲生成二次元老婆头像：

Figure 8: GAN生成的二次元头像

4 总结

GAN是一种应用广泛、潜力巨大的生成模型。它使用判别器和生成器进行对抗性训练，并最终产生足够逼真的图像。但GAN在训练过程中通常存在生成样本过于随机、训练不稳定等缺点，因此涌现出了诸多变体对其进行改进：CGAN、DCGAN、WGAN ……

我们期待在将来能看到更具潜力的GAN模型和更富价值的GAN应用！

Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014, June 10). Generative Adversarial Networks. https://arxiv.org/abs/1406.2661 ↩︎
事实上，这种“纳什平衡”是一种理想状态，实际训练难以达到。 ↩︎
关于梯度下降/上升法，可参考什么是梯度下降法？ - 马同学的回答 - 知乎。 ↩︎
Mirza, M., & Osindero, S. (2014, November 6). Conditional generative adversarial nets. https://arxiv.org/abs/1411.1784 ↩︎
Radford, A., Metz, L., & Chintala, S. (2016, January 7). Unsupervised representation learning with deep convolutional generative Adversarial Networks. https://arxiv.org/abs/1511.06434 ↩︎
令人拍案叫绝的 Wasserstein GAN - 郑华滨的文章 - 知乎 ↩︎
Jin, Y., Zhang, J., Li, M., Tian, Y., Zhu, H., & Fang, Z. (2017, August 18). Towards the automatic anime characters creation with Generative Adversarial Networks. https://arxiv.org/abs/1708.05509 ↩︎

雪

Mon, 11 Dec 2023 00:00:00 +0800

本来是不喜欢冬天的。

原来顶喜欢的季节是秋天。漫步街头，总能与一片飘然坠于身前的落叶不期而遇。爱极了这种意料之外的邂逅。

后来听过 luna 的あの夏のいつかは（在那個夏日的某天），夏天在我心中的地位渐渐变得与秋天不分伯仲了。每次看这首曲子的 pv，都深为夏的热烈激动不已：“正是无数个微小片刻，汇集在一起造就这热烈的我”。生命本该如此。

但是，提到“冬”这个字，浮现在脑海中的总是凛冽的风、凋零的树、蜷缩的人……讨厌这种万事万物都被压抑的感觉：生命似乎被严酷的冬冰封了起来，待到来年春暖花开的时节，才能重焕生机。冬天少了点灵魂。

后来才渐渐发现我错了。冬不是没有灵魂；相反，冬的灵魂甚至比其他三个季节都更加可爱——这正是那名为“雪”的精灵。雪洁白、轻盈、古灵精怪，冬天的沉闷因为她的到来一扫而空。“忽如一夜春风来，千树万树梨花开。”被雪点缀后的世界，焕发出不亚于春的勃勃生机。人们不再蜷缩在被窝，校园里随处可见飞舞的雪球、奇形怪状的雪人和创意百出的雪地上的图案。冬天原来是这样一个趣味盎然的季节。

独身骑行在清华园内，走走停停，停停走走，一时将诸多ddl抛在脑后，只觉心灵也因这白茫茫的雪的世界变得一片通透。终于明白原来没有一个季节是不值得喜爱的，生命中也没有一个时刻是不值得热爱的。人活在世，本该如此。

本该这样喜欢冬天啊。

Hello, World!

Sat, 09 Dec 2023 00:00:00 +0800

建好了博客，第一件事当然是发一篇 Hello,world! (bushi

~~实际上没想好要写啥，所以随便水一篇博客~~

放首DT：

Daucloud's Blog

减脂饮食笔记

TL;DR

起始配额

力量训练日

大体重

休息日或只做有氧

两周一调，只动碳水

训练日怎么吃

早餐

训练前餐

训练后餐

其他餐

按训练时间套用

休息日

有氧

关于 ALF 中 Bias 引起的 MoE 路由翻转的简要讨论

问题

TL;DR

对开源 checkpoint 的实际诊断

bias range 与翻转的确定性关系

checkpoint 中的 bias 有效范围

翻转情况的统计

不严谨的数学解释

b\mathbf{b}b 的对称初始化和负反馈更新

考虑负载均衡的平衡条件

Open Questions

恢复了旧博客！

RL Note 3: Markov Decision Process

Prologue

Introduction

Markov Decision Process

Bellman Equation

Bellman Equation In a Matrix Form

Conclusions

RL Note 2: Multi-Armed Bandits

Prologue

Problem Formulation

Regret

Definitions

Lower Bounds

Explore-Exploit Dilemma

MAB From the Perspective of RL

Tail Bounds2

Markov’s Inequality

Chebyshev’s Inequality

Chernoff Bound

Hoeffding’s Inequality

Explore-Then-Commit

Varaints of ETC

Epoch Greedy ETC

ε\varepsilonε-Greedy ETC

Upper Confidence Bound (UCB)

RL Note 1: Basics

Prologue

What is Reinforcement Learning?

Action vs. Reward

Definitions

What does ‘presented options’ mean?

The Goal of RL

State vs. Value

How does the agent take an action?

Definitions

What does the subscript π\piπ mean?

How to interpret the E\mathbb{E}E?

What does γ\gammaγ mean?

Policy

Model

Conclusion

Chezmoi——一种优雅的点文件管理工具

概述

安装

基本使用

在新机器上应用配置

模版

脚本

加密

杂项

使用Latex Worksop时，如何引入Minted宏包？

Hello World

$\mathbf{b}$ 的对称初始化和负反馈更新

Tail Bounds²

$\varepsilon$ -Greedy ETC

What does the subscript $\pi$ mean?

How to interpret the $\mathbb{E}$ ?

What does $\gamma$ mean?