Drift of GPT2 when reusing cache too much

drift.py

TEXT = """Mr. Phileas Fogg lived, in 1872, at No. 7, Saville Row, Burlington
Gardens, the house in which Sheridan died in 1814.  He was one of the
most noticeable members of the Reform Club, though he seemed always to
avoid attracting attention; an enigmatical personage, about whom little
was known, except that he was a polished man of the world.  People said
that he resembled Byron--at least that his head was Byronic; but he was
a bearded, tranquil Byron, who might live on a thousand years without
growing old.

Certainly an Englishman, it was more doubtful whether Phileas Fogg was
a Londoner.  He was never seen on 'Change, nor at the Bank, nor in the
counting-rooms of the "City"; no ships ever came into London docks of
which he was the owner; he had no public employment; he had never been
entered at any of the Inns of Court, either at the Temple, or Lincoln's
Inn, or Gray's Inn; nor had his voice ever resounded in the Court of
Chancery, or in the Exchequer, or the Queen's Bench, or the
Ecclesiastical Courts.  He certainly was not a manufacturer; nor was he
a merchant or a gentleman farmer.  His name was strange to the
scientific and learned societies, and he never was known to take part
in the sage deliberations of the Royal Institution or the London
Institution, the Artisan's Association, or the Institution of Arts and
Sciences.  He belonged, in fact, to none of the numerous societies
which swarm in the English capital, from the Harmonic to that of the
Entomologists, founded mainly for the purpose of abolishing pernicious
insects.

Phileas Fogg was a member of the Reform, and that was all.

The way in which he got admission to this exclusive club was simple
enough.

He was recommended by the Barings, with whom he had an open credit.
His cheques were regularly paid at sight from his account current,
which was always flush.

Was Phileas Fogg rich?  Undoubtedly.  But those who knew him best could
not imagine how he had made his fortune, and Mr. Fogg was the last
person to whom to apply for the information.  He was not lavish, nor,
on the contrary, avaricious; for, whenever he knew that money was
needed for a noble, useful, or benevolent purpose, he supplied it
quietly and sometimes anonymously.  He was, in short, the least
communicative of men.  He talked very little, and seemed all the more
mysterious for his taciturn manner.  His daily habits were quite open
to observation; but whatever he did was so exactly the same thing that
he had always done before, that the wits of the curious were fairly
puzzled.

Had he travelled?  It was likely, for no one seemed to know the world
more familiarly; there was no spot so secluded that he did not appear
to have an intimate acquaintance with it.  He often corrected, with a
few clear words, the thousand conjectures advanced by members of the
club as to lost and unheard-of travellers, pointing out the true
probabilities, and seeming as if gifted with a sort of second sight, so
often did events justify his predictions.  He must have travelled
everywhere, at least in the spirit.

It was at least certain that Phileas Fogg had not absented himself from
London for many years.  Those who were honoured by a better
acquaintance with him than the rest, declared that nobody could pretend
to have ever seen him anywhere else.  His sole pastimes were reading
the papers and playing whist.  He often won at this game, which, as a
silent one, harmonised with his nature; but his winnings never went
into his purse, being reserved as a fund for his charities.  Mr. Fogg
played, not to win, but for the sake of playing.  The game was in his
eyes a contest, a struggle with a difficulty, yet a motionless,
unwearying struggle, congenial to his tastes.

Phileas Fogg was not known to have either wife or children, which may
happen to the most honest people; either relatives or near friends,
which is certainly more unusual.  He lived alone in his house in
Saville Row, whither none penetrated.  A single domestic sufficed to
serve him.  He breakfasted and dined at the club, at hours
mathematically fixed, in the same room, at the same table, never taking
his meals with other members, much less bringing a guest with him; and
went home at exactly midnight, only to retire at once to bed.  He never
used the cosy chambers which the Reform provides for its favoured
members.  He passed ten hours out of the twenty-four in Saville Row,
either in sleeping or making his toilet.  When he chose to take a walk
it was with a regular step in the entrance hall with its mosaic
flooring, or in the circular gallery with its dome supported by twenty
red porphyry Ionic columns, and illumined by blue painted windows.
When he breakfasted or dined all the resources of the club--its
kitchens and pantries, its buttery and dairy--aided to crowd his table
with their most succulent stores; he was served by the gravest waiters,
in dress coats, and shoes with swan-skin soles, who proffered the
viands in special porcelain, and on the finest linen; club decanters,
of a lost mould, contained his sherry, his port, and his
cinnamon-spiced claret; while his beverages were refreshingly cooled
with ice, brought at great cost from the American lakes.

If to live in this style is to be eccentric, it must be confessed that
there is something good in eccentricity.

The mansion in Saville Row, though not sumptuous, was exceedingly
comfortable.  The habits of its occupant were such as to demand but
little from the sole domestic, but Phileas Fogg required him to be
almost superhumanly prompt and regular.  On this very 2nd of October he
had dismissed James Forster, because that luckless youth had brought
him shaving-water at eighty-four degrees Fahrenheit instead of
eighty-six; and he was awaiting his successor, who was due at the house
between eleven and half-past.

Phileas Fogg was seated squarely in his armchair, his feet close
together like those of a grenadier on parade, his hands resting on his
knees, his body straight, his head erect; he was steadily watching a
complicated clock which indicated the hours, the minutes, the seconds,
the days, the months, and the years.  At exactly half-past eleven Mr.
Fogg would, according to his daily habit, quit Saville Row, and repair
to the Reform.

A rap at this moment sounded on the door of the cosy apartment where
Phileas Fogg was seated, and James Forster, the dismissed servant,
appeared.

"The new servant," said he.

A young man of thirty advanced and bowed.

"You are a Frenchman, I believe," asked Phileas Fogg, "and your name is
John?"

"Jean, if monsieur pleases," replied the newcomer, "Jean Passepartout,
a surname which has clung to me because I have a natural aptness for
going out of one business into another.  I believe I'm honest,
monsieur, but, to be outspoken, I've had several trades.  I've been an
itinerant singer, a circus-rider, when I used to vault like Leotard,
and dance on a rope like Blondin.  Then I got to be a professor of
gymnastics, so as to make better use of my talents; and then I was a
sergeant fireman at Paris, and assisted at many a big fire.  But I
quitted France five years ago, and, wishing to taste the sweets of
domestic life, took service as a valet here in England.  Finding myself
out of place, and hearing that Monsieur Phileas Fogg was the most exact
and settled gentleman in the United Kingdom, I have come to monsieur in
the hope of living with him a tranquil life, and forgetting even the
name of Passepartout."
"""

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

full_input_ids = tokenizer.encode(TEXT, return_tensors="pt")
k = 20
for i in range(0, 300, 10):
    input_ids = full_input_ids[:, i : i + 1023]
    position_ids = torch.arange(0, 1023).unsqueeze(0)
    outputs = model.forward(input_ids=input_ids, position_ids=position_ids)

    invalid_position_ids = torch.zeros((1, 1023)).long() + 1022
    output_biased = model.forward(input_ids=input_ids, position_ids=invalid_position_ids)

    proposed_tokens_biased = set(output_biased.logits[:, -1].log_softmax(dim=-1).topk(k=k).indices.tolist()[0])
    proposed_tokens = set(outputs.logits[:, -1].log_softmax(dim=-1).topk(k=k).indices.tolist()[0])

    print(len(proposed_tokens.intersection(proposed_tokens_biased)) / k)