29 byte X

gistfile1.txt

ov = $0D ; == $FF, initial value for the overflow counter

* = $0304 - (end - scroll)

scroll: jsr $AAD7 
loop:   lda #$A0
        sta $D3E0 - $80, x
        sta $0400+40*24+39-$80+1, y     ; incrementing rower, y finishes with value $80
        sta $0400+40*24-$7B-1, x        ; decrementing rower, x finishes with value $7b
        adc ov
        sta ov
        dex
        iny
        bmi *
        bcc loop
        jmp scroll
end:

Doesn't jmp take three bytes? bcs would take two.
Ach, got an explanation from nurpax.

The jmp actually saves a byte, as the two parameters bytes re-use the address in $0302/$0303. This address contains the "BASIC idle" vector. After a program has been loaded, the C64 jumps to the address stored in $0302/$0303 (which is now pointing to the start of this program). A typical C64 auto-start mechanism.

Christ... I wrote a cycle accurate 65C02 processor emulator and I can't understand how you pulled this off! Bravo!

It's not just 6502/6510, it also relies on side effects of the KERNAL routines (I think $0d is loaded with #$ff as part of the C64 boot-up process) and it overwrites the $0302/$0303 vector, so the KERNAL never really returns from the LOAD to BASIC but starts the program instead)). It also spams many VIC memory addresses (the VIC appears several times in the memory map at "mirrored addresses"), eventually also hitting the correct addresses $d020/$d021 (which also appear at $d320/$d321) with #$a0 (which the VIC interprets as #$00 (= color code for black), as it only considers the lower four bits).
I guess you already figured that DEX/INY leave the carry bit alone, so the "overflow variable" can be used to determine whether the call to the scrolling routine needs to be executed or skipped.