上篇： 从零开始的 LLM: nanoGPT 学习笔记（1/2） 尝试了完整的训练的过程，nanoGPT 仓库中还有复现 GPT2 的代码，可惜对计算资源要求太高（基于 OpenWebText 数据集，8 卡 A100，训练 4 天），不是个人电脑玩的转了，只能跳过这一步，尝试后面的 finetuning。
finetuning
1. 训练数据
跟 pre-train 一样的 shakespeare 数据集，并按 9:1 切分 train/val：
# download the tiny shakespeare dataset
input_file_path = os.path.join(os.path.dirname(__file__), 'input.txt')
if not os.path.exists(input_file_path):data_url = 'https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt'with open(input_file_path, 'w', encoding='utf-8') as f:f.write(requests.get(data_url).text)with open(input_file_path, 'r', encoding='utf-8') as f:data = f.read()
n = len(data)
train_data = data[:int(n*0.9)]
val_data = data[int(n*0.9):]
区别在于 tokenization 使用 OpenAI 的 tiktoken 库， GPT-2 byte-pair encoding (BPE) tokenizer
# encode with tiktoken gpt2 bpe
enc = tiktoken.get_encoding("gpt2")
train_ids = enc.encode_ordinary(train_data)
val_ids = enc.encode_ordinary(val_data)
2. 开始训练
使用配置：init_from = 'gpt2' 基于 GPT-2 模型开始 finetune。
其实 finetune 的过程与 pre-train 并没有区别，无非是神经网络的参数是随即初始化的还是已经训练好的区别而已。
python train.py config/finetune_shakespeare.py
3. 推理效果
python sample.py --out_dir=out-shakespeare
执行完上面的代码，返回值为（太长，折叠起来了）：
点击查看代码
hi-shlaf-as-liq-al-mawww-al-shamsah(An'al-liqal, q.v.)I have a daughter.Lalal-lal-al-liq-al-shaf-al-lil-al-liq-al-qawww-al-al-shamsahI am a prophet and call Allah.Hak'l-al-mad-al-'al-bar'il-al-al-al-a'lisawahAllah is One.Lalal-al-al-al-al-al-al-al-al-alI have a son.Lalal-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-alI am an apostle and call Allah.Lal-al-al-al-al-al-al-al-al-al-al-al-alI have a son.Hak'l-al-al-al-'al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-al-
---------------
hi:You are a spy, who shall behold at my coming.Barracks:
You are the king, and I shall depart to search if you will.Barracks:
There are no spies, but, as for the king, he has her.Barracks:
There are spies in all, and I shall speak to you only.Barracks:
I have told her that my Lord is dead.Barracks:
Thus are informed, that he is now dead.Barracks:
You are dead, and my lord must die.Barracks:
Lord, you are dead, and my lord must die.Barracks:
I do swear to you.Barracks:
That is the way, and it is the law, and it is the law, and it is the law, and it is the law, and it is the law, and it is the law, and they are laws
And yet, I will not speak to you.Barracks:
Can you, say we?Barracks:
If not, let me have it for the king.Barracks:
The king has the crown and the crown is the king.Barracks:
The king is dead, there are no spies.Barracks:
There are no spies, but, as for the king, he has her.Barracks:
There are spies in all, and I shall speak to you only.Barracks:
There are spies in all, and I shall speak to you only.Barracks:
There are spies in all, and I shall speak to you only.Barracks:
There are spies in all, and I shall speak to you only.Barracks:
There are spies in all, and I shall speak to you only.Barracks:
There are spies in all, and I shall speak to you only.Barracks:
There are spies in all, and I shall speak to you only.Barracks:
There are spies in all, and I shall speak to you only.Barracks:
There are spies in all, and
---------------
hi:A, aye, ereon.Thy lords, ye have them to see: if ye do not, we are bound to, you know, shall not.Saith the lord:
Your lords,
O Sire, how I put you, and,
your lord, free and sober, thou let me know
that, the lord, is me.Thy lords,
I have given you the power.Thy lords,
I have my hand, and my house:
My father, my son, and my daughter:Nay, I have done what I have done;
I have done it, and this, and this, and this,
as I do and this as I have done.Your lords,
My father, and my son,And thy house, and thy noble house:
Forgive me my anger, if I persecute thee.And thy lords,
O my lord,
I have done what I have done,
with my eyes opened like a man:
I have not made a man a son:
In my power, and my power,
I have been my father's, his servants', and
his servants' servants' servants.Thy lords,
I have given thee the things thou wouldst have.And thy lords,
O my lord,
I have made thee thy fathers' and my sons'
and their daughters' and their sons'.And thy lords,
O my lord,
My lords, I have done them in the sight
of thee.My children, thou art,
Now, I am just, and thou art mine to-day.O, thou art,
Now, I am simply, and thy children
are not mine to-day, then, let me have thee
when thou shallest find me.My children,
I am not,
now, but they are the children of mine.O, thou art,
Now, I am but a little boy,
and now, I am but a little man.O, thou art,
Then, I am the son of thy father, and thy son
is not the son of thy father,
nor of thy son:
And I am the son of
---------------
hiHis name is SarwaHe is a prophetAnd is your father's servant: I
Treat him reverently and reverently all the time
That he is known.In that
Romeo,That
Sing
This is thy honor:
A very high
Commit.Thou art thy father:
Here is thy father,
The most famous of men in Rome;
I have never heard him speak thus to me.
But if it be so, I am in his bed.
Thou art thy father's servant:
And from thy bed there is thy dream.
Now we shall make thee
a gentleman.Yours truly,I hast never seen
Thou thy father thus spoken to me.Will this be too much trouble for the Romans?I do not think it very much trouble.Tell me once or twice,
I pray thee good luck:
Do not be troubled with the maid-servant,
If this is in thy sight.And whose duty is it to the maid-servant?I am to the queen of Rome,
To lead her to the Roman throne.We have been together in heaven: we are
The natural sons of the Lord,
And we are of good taste.And thus,
I answer thy good words.I am not persuaded,
In thy hearing,
That thou shalt am the best.If I sent the prince of the Roman city,
To speak to the maid-servant,
To speak to her,
To speak to her, I can persuade thee,
That this is thy good honour.What is it said,
Thou old man of the court?For thou art the greatest of men to me:
I am no less worthy of thy dignity.Madam,
I would be so very much worse than thou,
If I had not made a right choice,
If thou art the most gentle,
If thou art the most gracious;
If thou art the most eloquent,
If thou art the most temperate,If thou art the most beneficent;If thou art the most sublime,
If thou art the most sublime,
If thou art the most sublime,
---------------
hi in the past that you are of good faith.The night had not come and the maids had been gone all night.Misty misty slumber, alas.These are the voices of the maids,
And they are the voices of the maids
To the maids of the rain and the misty:
The maids are the maids.'She was so glad,
Which she had'd to make of the maids,
But to do with the maids.The maids say to nature: 'She's gone out of the cupboard,
And our day.'Here is the fair maid who is hungry but not yet gone,
And here is the fair maid who is mad and not yet gone,
And this she's brought home
On the other side.Misty misty slumber, alas.The maids call on the maids,
Which say they are not all well.They have left the maids at the old house.Misty misty slumber, alas.The maids' go to Iolanda,
And the maids' go to Corriere,
And the maids' go to the fair maids.Misty misty slumber, alas.The maids' go to Corriere,
And these maids' go to the fair maids.Misty misty slumber, alas.The maids' go to Corriere,
And the maids' go to the fair maids.Misty misty slumber, alas.The maids' go to Corriere,
And these maids' go to the fair maids.Misty misty slumber, alas.The maids' go to Corriere,
And these maids' go to the fair maids.Misty misty slumber, alas.The maids' go to Corriere,
And these maids' go to the fair maids.Misty misty slumber, alas.The maids' go to Corriere,
And these maids' go to the fair maids.Misty misty slumber, alas.The maids
---------------
hi, who is an old man of these two, and very long. How much more now we have the way come; but it is as the time of thy glory; and these we are.TOLSTOYNFor the end, what end shall the world be.MARSHALLO, he hath an end.THE LORDNow, let him do what he will thereby.TOLSTOYNThe Lord, pray pardon, sir, let him, O Lord, come, and call forth the light of the church; and let him choose a place to go and hear it.THE LORDGod bless him.MARSHALLThis is as it are.THE LORDFor I pray thee to thy gracious mercies.TOLSTOYNGod help thee, O Lord, and showth with exceeding grace thee this night.MARSHALLO, and read the words of thy servant;
O God thy spirit hath been made holy;
As thou hast made thyself holy, so hath thy spirit helped thee.THE LORDLet him be the light of the church, and let him go where he will.THE LORDLet him go from thy house.THE LORDOur holy servant.CAMBRIDGEHow much more, how much more, till thou art thy husband's end.MARSHALLO, and let him go hither.THE LORDO, and let him go away.CAMBRIDGEO, let him go hither.THE LORDWhat thou hast done, go out to Christ thy Lord.CAMBRIDGEO, let him go hither.THE LORDBut let him go along.CAMBRIDGEO, let him go hither; thus shalt thou die.THE LORDO, and thou shalt give him a death, for he hath gone, and hath laid his soul out as a mark,
and hath been here to kill it:
and it is not good to come unto him.CAMBRIDGEO, let him go hither.THE LORDLet
---------------
hion to the end of the day, and with all sincerity to the end of a life. 'Shall I not do but the Lord will grant of his grace and grace to our wife's bosom unto the end of the day?' 'We have not prayed for an hour,' quoth the Lord: 'if we do, the Lord will grant of our grace, and therefore with the holy Spirit he will grant my grace. I look forward to the end of the day to meet the Lord. I shall bless his house and he shall bless mine.'11.Deuteronomy 19:9-10The Lord says to the servant-servant, 'Father, let us eat and drink and not be with thee, and let us be children.'The servant-servant, ' you shall not be with thee, ''Let us live and be children.'Til now we have made you an offering without sin,
And when we are given, thou hast given us food and drink.'If thou art come hither, we shall be buried in thy tomb.'Why then, therefore, say to the servant-servant, 'I am come hither with thee, and shall be buried in thy tomb.''In thy tomb!' says the servant-servant. 'May our prayers and our joy be with thee.'12.Deuteronomy 22:3-4As the Lord says unto the servant-servant, 'Be with me,' says the servant-servant. 'I am with thee, and am not with thee any more. But and of your fathers, thou art thy father and mother.
And my father, ' thou are my father.'
And thy mother, ' thou are thy Mother.'Then the servant-servant says unto him, 'Lord, I have not been with thee many days.
'I have taken thee by the hand, and I am with thee several times.
And now, when I am come, let us pray for thy father and mother.
And now, 'tis a great grace to worship you and take thy mother.'
And now, 'tis a great grace to worship thee and to pray for thy mother.'Then the servant-servant says unto him, 'Father, let us pray for thy father and mother.
'
---------------
hiS, O.I.O.S.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,S.P.E.H.M.L.I.A.R,
---------------
hi is the queen of the princely land, the son of a Christian king, who lives in two parts of the land and makes a king.When you were born he had a son, who lived on his father's side. He hewed him a son by the time he was twelve years old.When I had been in his house when he was twelve years old, he was in an arms field, and he made the king a king, and the king made his father a king.Then the king hewed him a son by the time he was fourteen years old, and the king hewed him another son by the time he was twelve years old.And how was it done, and what was the matter?I thought you may inquire at the time.I am careful to set you in good stead,
but at the time you are set in what you are set in,
you will do well.CHAPTER XXI.Then after your father's death, you go to the king's house, and there make a treaty,
and he gives you a horse.So we all go and greet the king.MYTHBOUY:
My lord, there is a lie in your heart,
That that is a tale told, is a lie told.But it is an old tale: and I know it
Even as I know its truth.And you mean your mother's daughter,
With whom your father was much in love.You have deceived me, and it was by your looks
That you were deceived.Nay, it is far too great a lie,
That a lie must lie between two men.Your son or his daughter has been deceived by this.Nay, trust me, you lie.MYTHBOUY:
There is a lie in that which I am deceived by,
that you have it with the woman,
Which you have been deceived by the woman.And I know it is a lie, too, even as I know the truth.My son, it is a lie, too, although with the woman in order
To deceive her.YOUR KING:
Why is it called to you that I am deceived?Nay, I am your father, my mother.---------------
hi on:The whole class of God is one, but one Christ do not see.The entire class of God is one, but one Christ.The whole class of God is one, but one Christ.God there be:
I am come not unto this world,
I do not come unto this world,
I am come not to this world.God there be:
Come ye, behold, and observe that all that be here have faith in me
and do these things, that I may make known that I am in this world,
and that you may dine with me, make with me
you that I may make known my nature, and that you may dine with me.To all:
I am not come not to this world;
that is to say no one can come here, but only one.God there be:
Why, what do you think?Did you say that I am come not to this world?
Were you not so bold as we are to be seen
to this world and to our seeing? Were you not so bold as we are to be seen?
To all:
I am come not unto this world,
I am come not unto this world;
I am come not unto this world:
I am come not unto this world:
I am come not unto this world:
I am come not unto this world:God there be:
What are we, that we may look for the resurrection of their dead?God there be:
I am come not unto this world,
I am come not unto this world:
I am come not unto this world:God there be:
I am come not unto this world:
I am come not unto this world:God there be:
What have I done that is for you?God there be:
I am come not unto this world;
I am come not to this world:God there be:
I am come not unto this world:God there be:
What have I done that is for you?God there be:
I am come not unto this world:I am come not to this world:God there be:
What have I done that is for you?God there be:---------------
从结果看，有了 hi, who is an old man of these two... 这种明显的成句的补全，总体生成的字符也增长到了 14000+（我们自己训练的模型，10次推理总共只有 5000+ 的字符）。
一开始没搞明白为啥自己 pre-train 的模型，每次只有 502 个字符（初始的 hi 加上配置 max_new_tokens = 500，一共 502），而基于 GPT-2 微调的模型，则每次都有 1000+ 的字符，而且数量不等，后来才反映过来，自己 pre-train 的时候 tokenization 按字符来的，而 GPT-2 是使用的是 BPE tokenizer，token 数量是一样的，那字符数量肯定不一致了...
