"Problems with nml"

gistfile1.txt

“The trouble with nml”

52:57  <andythenorth> to my mind, the CPP approach of dumping constants into one global namespace
06:53:02  <andythenorth> is a mess
06:53:15  <andythenorth> but…it makes for very compact templating :P
06:53:42  <andythenorth> no obj.obj.property or obj.method(obj.property) stuff
06:54:06  * andythenorth removing CPP from FIRS slowly
06:56:11  <Alberth> it's not designed to do large scale complicated replacements
06:56:37  <andythenorth> it’s a powerful tool that goes a long way fast :)
06:56:49  <andythenorth> but when it hits the limit, it hits it quite hard
06:57:38  <Alberth> it is, but "working" and "the correct approach" are two different things, which are sometimes seemingly well aligned
06:58:34  <Alberth> I agree it's better to do without CPP, Python is a much more capable solution
06:58:39  <Alberth> at the cost of more work
06:58:59  <andythenorth> removing it from here is quite tricky http://dev.openttdcoop.org/projects/firs/repository/entry/src/templates/produce_secondary.pypnml#L15
06:59:13  <andythenorth> all of the ‘var_leftover_cargo_1’ etc are CPP defines, mostly global
06:59:34  <andythenorth> it’s technically trivial to do, but will make word soup :P
07:00:32  <Alberth> I'd keep it, probably
07:00:39  <andythenorth> var_leftover_cargo_1 -> ${industry.get_perm_storage_num[‘var_leftover_cargo_1’]}
07:00:50  <andythenorth> does not aid reading a complicated templated switch :)
07:01:21  <Alberth> yes, if you go all the way, things become unreadable
07:02:09  <Alberth> you could introduce new names in your ${} thingies
07:02:24  <Alberth> to make the templates more readable
07:02:59  <Alberth> but generated code is barely readable, and pretty much non-modifiable
07:03:25  <Alberth> keeping the #define improves things at that level
07:04:30  <Alberth> In C++, the general direction is to eliminate #define, and replace with "inline" functions and "const" variables
07:04:45  <Alberth> ie the C++ languages itself provides safer alternatives
07:06:13  <andythenorth> chameleon has template local defines, so they’re in the root namespace within a specific template block
07:06:14  *** DDR has quit IRC
07:06:33  <Alberth> I once looked into NML having better templates, but it became very messy very fast
07:06:33  <andythenorth> eg define=“var_leftover_cargo_1” and then use it as ${var_leftover_cargo_1}
07:06:41  <Alberth> too many hacks around the parser
07:06:48  <andythenorth> yes
07:07:02  <Alberth> local defines look nice
07:07:20  <andythenorth> they’re generally frowned up by chameleon users, it creates annoying local indirection
07:07:36  <andythenorth> it’s preferred to put them in the backing view / object
07:07:38  <andythenorth> but eh
07:08:02  <Alberth> and local indirection is bad for computation costs?
07:08:12  <andythenorth> bad for readability
07:08:17  <Alberth> ah, ok
07:08:33  <andythenorth> if FIRS production code was simpler, this problem would go away :D
07:08:46  <Alberth> you have to be careful what to #define as well :)
07:09:45  <Alberth> although at the point of using ##, I think you're a few miles beyond that :)
07:10:15  <andythenorth> :P
07:10:52  <Alberth> I think to some extent, NML code isn't really designed from the higher level
07:11:17  <Alberth> it basically mirrors the action codes, and leaves combining the primitives to the user
07:12:05  <andythenorth> I find it easy to template, because it’s exactly akin to templating html
07:12:11  <andythenorth> which has the same issue
07:12:27  <Alberth> perhaps the higher level isn't really possible, as you'd cut of parts of the available design space
07:12:29  <andythenorth> but the techniques for good html templating took 10 years to develop :P
07:13:02  <Alberth> ie you'd end up with grfmaker-ish solutions, in the extremely simple-ish case
07:13:22  <andythenorth> yes: at first glance it looks easy to move NML up one level.  Could define a vehicle object etc, and hide away all the code
07:13:30  <andythenorth> but then the varaction2 chains....
07:14:43  <andythenorth> I know how I’d re-design it, but I’m fairly certain I’d be wrong
07:15:35  <Alberth> tbh, I don't think 'switch' is a right primitive, an if-tree would be nicer
07:15:50  <Alberth> computer can generate switches
07:15:58  <andythenorth> I find the switch syntax non-intuitive
07:16:27  <Alberth> yep, it maps directly on action2 sequences
07:16:45  <andythenorth> tbh, if the complex FIRS switches hadn’t been written by pm, yxo, hirndo etc…I’d never have grokked how to do them
07:16:54  <Alberth> no code generation required there :)
07:17:46  <andythenorth> expressions [] have remarkable power :P
07:17:59  <Alberth> I can see that, there is a lot of combined functionality hidden in the switch
07:18:47  <Alberth> but you'd want to write code more like plain imperative programming, eg Python function or C statements
07:19:32  <andythenorth> +1
07:19:39  <andythenorth> it’s also a bit heavy on syntax
07:19:43  <andythenorth> ‘said the python programmer'
07:20:07  <Alberth> yeah, too much functionality packed in it
07:20:55  <andythenorth> putting aside the syntax…
07:21:09  <andythenorth> instead of FEAT_XXX and id
07:21:20  <andythenorth> I would rather have wrapped entire blocks of nml in a scope
07:21:27  <andythenorth> e.g. specific vehicle, industry, tile etc
07:22:08  <andythenorth> this actually mirrors how most of nfo works when it comes to variable access
07:22:29  <andythenorth> and it means the objects could be extend with props that aren’t in the nfo spec
07:22:48  <andythenorth> and trivial templating could then be used
07:23:34  * andythenorth should make a pate
07:23:35  <andythenorth> paste *
07:25:03  <Alberth> makes sense
07:25:43  *** zeknurn has quit IRC
07:30:16  <andythenorth> https://paste.openttdcoop.org/pgzr2xfsv
07:31:03  <andythenorth> it localises all switches etc; there would need to be a method to access global switches, as that’s an nfo feature which is occasionally very useful
07:37:04  *** tokai has joined #openttd
07:37:04  *** ChanServ sets mode: +v tokai
07:39:38  <Alberth> you just move FEAT_ROADVEHS up one level? or are the switch names now also limited in this scope?
07:39:47  <Alberth> (and other names, I guess)
07:40:05  <andythenorth> switch names are limited in this scope
07:40:14  <andythenorth> there would have to be a way to break out of that
07:40:25  <andythenorth> but eh, most people would never need it, especially in vehicle sets
07:40:37  <andythenorth> and people like MB who understand all this…have their own solutions
07:41:11  <andythenorth> I wonder if it would be more performant, because there was a half-idea that figuring out switch IDs in a global namespace is slow
07:41:22  <andythenorth> I dunno if that was profiled though
07:41:47  <Alberth> slow in nml, you mean?
07:42:12  <Alberth> wouldn't expect so, tbh
07:42:31  <Alberth> nml gets killed by too much copying of data
07:43:31  * andythenorth did consider going back to nfo, and templating it with chameleon :P
07:43:38  <andythenorth> it would be maybe 5x faster to compile
07:44:05  <Alberth> quite possibly a lot faster :)
07:44:10  *** tokai|noir has quit IRC
07:45:32  <andythenorth> somewhere my idea gets paired with simple ${name} templating
07:45:41  <andythenorth> but I can’t figure out why that’s useful
07:46:18  <Alberth> you may end up with something like m4nfo
07:46:26  <andythenorth> yup
07:46:30  <Alberth> which is basically also just a big set of templates
07:47:31  <andythenorth> ah, interesting.  The majority of my chameleon use is something like ‘${vehicle.id}_some_switch_name’
07:47:36  <andythenorth> because all switches must be named :P
07:47:44  <andythenorth> the scope idea eliminates that need
07:48:13  <Alberth> it would indeed
07:48:52  *** zeknurn has joined #openttd
07:48:57  <Alberth> it would also fix the need for ## in CPP :p
07:49:39  <andythenorth> e.g. http://dev.openttdcoop.org/projects/road-hog/repository/entry/src/templates/capacity_switches.pynml
07:50:00  <andythenorth> 7 uses of ${stuff} just to create switch ids
07:50:10  <andythenorth> 10 even
07:50:33  <andythenorth> 7 other uses
07:50:56  <andythenorth> ha if nml did ${some python code}
07:51:06  <andythenorth> then the parser could call eval() on it
07:51:12  <andythenorth> no risk there :P
07:52:18  <Alberth> there are ideas in newer pythons for doing such things with string formatting
07:52:43  <andythenorth> you mean Template(), or something else? o_O
07:56:44  *** czaks has quit IRC
07:57:00  <Alberth> https://www.python.org/dev/peps/pep-0498/
07:58:06  <andythenorth> interesting :)
08:01:07  <andythenorth> still not sure why templating is useful inside vanilla nml
08:01:40  <andythenorth> now wondering if nml-native templating would make life more complicated for people applying their own templates
08:01:50  <Alberth> it eliminates CPP templating
08:02:33  *** czaks has joined #openttd
08:03:00  <Alberth> ie no need for CPP any more, thus no mingw
08:03:01  <andythenorth> in my example template here, nearly all the templating relates to managing IDs https://www.tt-forums.net/viewtopic.php?p=993849#p993849
08:03:08  <andythenorth> which would be eliminated by a scope
08:05:11  <Alberth> local/global could be done by defining global names outside any scope
08:05:20  <andythenorth> yes
08:06:55  <andythenorth> I’m not sure whether nml should go an extra step
08:07:22  <andythenorth> and introduce prototypes
08:08:01  <andythenorth> e.g. prototype (ROAD_VEHS, town_bus) {  [properties and switches] }
08:08:18  <Alberth> isn't that just a template?
08:08:21  <andythenorth> yes it is
08:08:24  <andythenorth> precisely
08:08:43  <andythenorth> I would expect to handle all that myself, using my preferred templating tools
08:08:57  <andythenorth> but it might aid people who write out a lot of nml long-hand
08:09:10  <Alberth> add template facility to nml, and $someone can make a set of such templates
08:09:11  * andythenorth thinks of the FIRS forks
08:09:32  <andythenorth> GarryG and spiff are doing it the hard way, by modifying the nml output :P
08:09:38  <Alberth> yeah, your code generation is likely very much magic to a lot of people
08:09:38  <andythenorth> it’s not even nicely formatted for them :P
08:09:51  <andythenorth> 3iff / spiff /s
08:11:33  <Alberth> Well, it's easy to understand, it just takes time to change things
08:11:53  <andythenorth> it’s yak-shaving :)
08:11:54  <Alberth> We have not that much patience, so we resort to magics :)
08:12:16  <andythenorth> they get bugs because they are manually maintaining 40 identical switches :)
08:12:50  <Alberth> sure, and without even the basic understanding of what things do
08:13:17  <Alberth> so I think it's amazing they get anything done
08:13:35  <Alberth> not sure if a template would improve things there
08:14:10  <Alberth> I mean, if you don't know NML, adding a template language layer on top of it would improve things?
08:14:17  <andythenorth> it requires the understanding of what’s common and what’s not
08:14:38  <andythenorth> and if the templates can’t be broken up further into sub-templates / macros, then it’s hard to get proper reuse
08:14:39  <Alberth> somewhat, I think NML could do stuff there as well
08:14:48  <andythenorth> I think it’s hard to win
08:15:02  <Alberth> nfo is a complicated language
08:15:29  <Alberth> much more complicated than eg C or Python
08:15:50  <Alberth> all kinds of dedicated purpose constructs
08:16:06  <andythenorth> scopes, I see no downside; nml-native templating, might be a tarpit
08:16:27  <Alberth> nml-native templating just kills CPP, nothing more
08:16:54  <Alberth> and for people like you, it has no value, as you prefer your own stuff
08:16:58  <andythenorth> exactly
08:17:10  *** supermop_home has joined #openttd
08:17:28  <andythenorth> constant-subsitution (with local / global scope), might be useful
08:17:34  <Alberth> I wonder though, how much CPP would you need with scopes?
08:17:34  <andythenorth> it it can be made easy to understand
08:17:50  *** czaks_ has joined #openttd
08:18:01  <Alberth> it would mostly become a sequence of #include
08:18:02  *** czaks has quit IRC
08:18:05  <andythenorth> CPP tends to be useless for the id stuff, it requires variadic macros which are bad
08:18:35  <andythenorth> I think CPP is mostly being used for #include and constant substitution
08:19:44  <Alberth> so you'd need a "const myvalue = 1+3;" like thing?
08:19:46  <andythenorth> constant substitution is helpful for things like “self.label_refits_allowed = ['FRUT','WATR’]” where that is shared by 10 vehicles
08:20:09  <andythenorth> self.label_refits_allowed = {global_edibles_tankers}
08:20:15  <andythenorth> or something
08:20:23  *** supermop_ has joined #openttd
08:21:35  <Alberth> const edibles_cargoes = ['FRUT', 'WATR'];    self.label_refits_allowed = edible_cargoes;
08:21:45  <andythenorth> yes
08:22:03  <andythenorth> w.r.t includes, I am agnostic
08:22:34  <andythenorth> I would always prefer to write my own compile, so I’m not a good person to specify
08:22:55  <andythenorth> but actually the #include format seems pretty straightforward, as long as circular refs are banned
08:23:32  *** supermop has quit IRC
08:23:42  <Alberth> CPP allows them, but runs out of stack space for circular references :)
08:24:09  <andythenorth> ha
08:24:22  <andythenorth> hmm, #include for e.g. an entire vehicle in one file is intuitive
08:24:27  <andythenorth> with one master file that assembles them
08:24:44  <andythenorth> it could also be used for fragments of code, by people who make that leap
08:24:54  <andythenorth> it wouldn’t be formalised templating, nor would it break
08:24:58  <Alberth> gives very loooooong error messages ("out of stack at line x of file bla, included from line y of file bla .....")
08:25:29  <andythenorth> so the problems seem to be:
08:25:33  <andythenorth> - switch syntax
08:25:40  <andythenorth> - managing IDs (yak-shaving)
08:25:47  <andythenorth> - constant substitution
08:25:49  <andythenorth> - templating
08:25:52  <andythenorth> - file inclusion
08:26:32  *** supermop_home has quit IRC
08:26:55  <Alberth> quite
08:27:09  *** Biolunar has joined #openttd
08:28:20  <andythenorth> templating I think is non-winnable
08:28:26  <andythenorth> and I am on the fence about file inclusion
08:29:26  <Alberth> is pretty simple to hook into the parser, normally
08:29:46  <Alberth> except it's currently useless, as CPP does it all
08:30:04  <Alberth> and current parser of NML has too much magic
08:30:17  <andythenorth> I find it magical :)
08:30:26  <andythenorth> the rest of nml is pretty easy
08:30:55  <andythenorth> it’s just a giant set of substitutions :P
08:34:04  <Alberth> :o  I have the reverse experience :)
08:34:20  <Alberth> I can see how the parser works, everything else looks like magic :)
08:34:55  <andythenorth> I understand conceptually that it must build a tree of objects





---- vanilla NML ----

switch (FEAT_ROADVEHS, SELF, acton_switch_loading_speed_by_cargo_0, cargo_classes & bitmask(CC_MAIL, CC_ARMOURED)) {
    bitmask(CC_MAIL): return 5;
    bitmask(CC_ARMOURED): return 5;
    return 3;
}

switch (FEAT_ROADVEHS, SELF, acton_switch_loading_speed_by_cargo_1, cargo_classes & bitmask(CC_MAIL, CC_ARMOURED)) {
    bitmask(CC_MAIL): return 7;
    bitmask(CC_ARMOURED): return 7;
    return 4;
}

switch (FEAT_ROADVEHS, SELF, acton_switch_loading_speed_by_cargo_2, cargo_classes & bitmask(CC_MAIL, CC_ARMOURED)) {
    bitmask(CC_MAIL): return 9;
    bitmask(CC_ARMOURED): return 9;
    return 5;
}

switch (FEAT_ROADVEHS, SELF, acton_switch_loading_speed, param_adjust_vehicle_capacity) {
    0: acton_switch_loading_speed_by_cargo_0;
    1: acton_switch_loading_speed_by_cargo_1;
    2: acton_switch_loading_speed_by_cargo_2;
}

switch (FEAT_ROADVEHS, SELF, acton_create_visual_effect, 0) {
    return 0;
}

item(FEAT_ROADVEHS, acton, 600) {
    property {
        name:                           string(STR_NAME_ACTON, string(STR_NAME_SUFFIX_COACH));
        cargo_capacity:                 27;
        sprite_id:                      SPRITE_ID_NEW_ROADVEH; //enable new graphics - nml constant
        introduction_date:              date(1990,01,01); // consist just supplies intro year - openttd randomises intro dates a bit anyway
        power:                          360hp;
        speed:                          90mph;
        weight:                         8.0ton;
        tractive_effort_coefficient:    0.7;
        cost_factor:                    44.3;
        running_cost_base:              RUNNING_COST_ROADVEH;
        running_cost_factor:            88.6;
        model_life:                     VEHICLE_NEVER_EXPIRES;
        vehicle_life:                   40;
        reliability_decay:              20; // default value
        retire_early:                   -10;
        sound_effect:                   SOUND_BUS_START_PULL_AWAY;
        refit_cost:                     0; // this needs to be 0 if we want autorefit without using cb
        refittable_cargo_classes:       bitmask(CC_PASSENGERS);
        non_refittable_cargo_classes:   bitmask(); // don't set non-refittable classes, increases likelihood of breaking cargo support
        cargo_allow_refit:              [];
        cargo_disallow_refit:           [];
        default_cargo_type:             PASS;
        cargo_age_period:               370;
        misc_flags:                     bitmask(ROADVEH_FLAG_2CC,ROADVEH_FLAG_AUTOREFIT); // nml constants
        length:                         7;
        effect_spawn_model:             EFFECT_SPAWN_MODEL_DIESEL;
    }
    graphics {
        cargo_capacity:                 acton_switch_cargo_capacity;
        purchase_cargo_capacity:        acton_switch_cargo_capacity;
        loading_speed:                  acton_switch_loading_speed;
        default:                        acton_switch_graphics;
        purchase:                       acton_sg_purchase;
        create_effect:                  acton_create_visual_effect;
    }
}

if (param[1]==0) {
    item(FEAT_ROADVEHS, acton, 600) {
        property {
            climates_available:             ALL_CLIMATES;
        }
    }
}


---- some form of scoping ----

scope(FEAT_ROADVEHS, acton, 600) {
    switch (SELF, switch_loading_speed_by_cargo_0, cargo_classes & bitmask(CC_MAIL, CC_ARMOURED)) {
        bitmask(CC_MAIL): return 5;
        bitmask(CC_ARMOURED): return 5;
        return 3;
    }

    switch (SELF, switch_loading_speed_by_cargo_1, cargo_classes & bitmask(CC_MAIL, CC_ARMOURED)) {
        bitmask(CC_MAIL): return 7;
        bitmask(CC_ARMOURED): return 7;
        return 4;
    }

    switch (SELF, switch_loading_speed_by_cargo_2, cargo_classes & bitmask(CC_MAIL, CC_ARMOURED)) {
        bitmask(CC_MAIL): return 9;
        bitmask(CC_ARMOURED): return 9;
        return 5;
    }

    switch (SELF, switch_loading_speed, param_adjust_vehicle_capacity) {
        0: switch_loading_speed_by_cargo_0;
        1: switch_loading_speed_by_cargo_1;
        2: switch_loading_speed_by_cargo_2;
    }

    switch (SELF, create_visual_effect, 0) {
        return 0;
    }

    property {
        name:                           string(STR_NAME_ACTON, string(STR_NAME_SUFFIX_COACH));
        cargo_capacity:                 27;
        sprite_id:                      SPRITE_ID_NEW_ROADVEH; //enable new graphics - nml constant
        introduction_date:              date(1990,01,01); // consist just supplies intro year - openttd randomises intro dates a bit anyway
        power:                          360hp;
        speed:                          90mph;
        weight:                         8.0ton;
        tractive_effort_coefficient:    0.7;
        cost_factor:                    44.3;
        running_cost_base:              RUNNING_COST_ROADVEH;
        running_cost_factor:            88.6;
        model_life:                     VEHICLE_NEVER_EXPIRES;
        vehicle_life:                   40;
        reliability_decay:              20; // default value
        retire_early:                   -10;
        sound_effect:                   SOUND_BUS_START_PULL_AWAY;
        refit_cost:                     0; // this needs to be 0 if we want autorefit without using cb
        refittable_cargo_classes:       bitmask(CC_PASSENGERS);
        non_refittable_cargo_classes:   bitmask(); // don't set non-refittable classes, increases likelihood of breaking cargo support
        cargo_allow_refit:              [];
        cargo_disallow_refit:           [];
        default_cargo_type:             PASS;
        cargo_age_period:               370;
        misc_flags:                     bitmask(ROADVEH_FLAG_2CC,ROADVEH_FLAG_AUTOREFIT); // nml constants
        length:                         7;
        effect_spawn_model:             EFFECT_SPAWN_MODEL_DIESEL;
    }
    if (param[1]==0) {
        property {
            climates_available:             ALL_CLIMATES;
        }
    }
    graphics {
        cargo_capacity:                 switch_cargo_capacity;
        purchase_cargo_capacity:        switch_cargo_capacity;
        loading_speed:                  switch_loading_speed;
        default:                        switch_graphics;
        purchase:                       sg_purchase;
        create_effect:                  create_visual_effect;
    }

} // end scope