Thoughts on Django’s form rendering

form-rendering-thoughts.md

Django's template-based form rendering: thoughts and pains

Short (well, I tried) collection of thoughts and impressions after moving to Django-native template-based form rendering with pretalx, my ~medium-sized(?) Django project.

When Carlton threatened to read my code (shock, horror), I decided to just write up my impressions, and a gist/pastebin/etc seemed the right format, cuz this isn’t polished enough for a blog post, and also way not constructive enough.

Context

In my main project pretalx (and therefore basically all toy projects, too) I've been using libraries like django-bootstrap4 to render my forms. This dates back to Django ~1.10 for me, so don’t judge!

django-bootstrap4 works by providing template tags for your forms. The most relevant tags are {% bootstrap_form form %}, {% bootstrap_field form.field } and {% bootstrap_form_errors form %}. You can pass additional parameters to modify CSS classes on wrappers / inputs, but I rarely used those.

The only modifications I had made to django-bootstrap4 was in how labels were rendered (as I wanted to include an "(optional)" behind optional field labels rather than a star/similar for required fields). The modification was on the hacky/annoying side, but I didn't have to touch them again, so that was alright.

The only modifications I had made to Django's form-rendering up to that point was to override django/forms/widgets/input_option.html because I don't like wrapping inputs in labels. I don't really like having that modification around, because if Django makes improvements to how it renders labelled input options, I won't get those improvements, but ehh, it's fine.

Motivation

I had stopped updating django-bootstrap4 due to frustrating update experiences – tiny modifications that you wouldn't notice right away, or annoying breakage of your form styles everywhere. I also made use of only a small subset of the library's functionality: django-bootstrap4 also provides templatetags to generate alerts, formsets, buttons, messages, pagination, custom widgets, etc.

I had also recently removed other big old dependencies from pretalx, namely jQuery and bootstrap4 itself (while retaining the classes and parts of it that I actually used, for a lighter and more readable frontend library). So I had no real reason to keep django-bootstrap4 around.

Implementation

You can see the full PR here, now in sensible commits. Notable to the discussion:

• Implementing the form renderers: This is the pleasant part. Just three templates for three different form presentations: Tabular, inline with labels, inline without labels.
• Removing bootstrap_form and bootstrap_field: The monster. Shows the bulk part of the work: Mostly straight replacement, some simplification, few workarounds.
• Using a base form template for trivial forms: In the big CRUD part of the application, some forms are really just {{ form }} with form tags and a submit button. I wrote a tiny base form template, and re-used it as much as I could. This is the part that felt really good.
• Less relevant, but a nice exercise in moving to regular form rendering: Bootstrap has the concept of input groups, where you add some text (or any other element) directly next to a text input in order to provide context or further actions. django-bootstrap4 provided this in their field template tag, but re-implementing it as a custom widget was painless and arguably improved the code.

Notes, grumps, feelings

tl;dr: The whole experience felt uneven and hard to do “right”. It's not obvious why Django provides some conveniences and not others (and I know that Django is resistant to adding any new features, so what we have is probably what we’ll get). It’s also not obvious where some things happen, and where they should happen, and the docs … well, I got most of my information from just reading the Django source code, which feels like it’d’ve been a big hurdle back when I got started.

In particular:

• I feel like Django should provide a base form template ({{ form.as_post_form }}?) for the standard POST form with a csrf_token, with enctype=multipart/form-data if needed, and a submit button. Maybe I'm over-generalising, but man, these parts are everywhere, and it’s so easy to lose time to debugging a missing CSRF token or the incorrect enctype. Sure, doesn’t happen to me much anymore, but that’s because I put in my time in the form template debugging mines.
  • As you saw above, I built this for myself with a template that I include and that I can pass context to (like the form, of course, but also the submit button value and label). But it’s not obvious that this is the way to do it – given the choice between {% include "base_form.html" %}, {{ form|base_form_wrapper }}, and {% base_form_wrapper form %}, it’s not obvious (especially if you have less Django experience!) which way to go. (Filters aren’t the way, but you probably only learn that the first time you wished you could pass better and more context to a filter, or read a bunch of prior art). It’s tempting to look at Django for what to do, but that’s not helpful – looking at Django is how I got my wish for {{ form.as_post_form }}, but we can't just add methods like that via our renderer – we’d have to add them to RenderableFormMixin, which isn’t very extensible.
• Maybe what I was really missing was more direct guidance on implementing your own form renderer / using template-based form rendering. For example, I got really hung up on rendering errors when you don’t use {{ form }}. The documentation notes that you can override error rendering by setting the renderer (not: default_renderer) on the ErrorList, but it’s not clear when (or if ever) you’d do this. Actually, I didn’t understand the implications of this at all at first – I thought I had to do something like {% include "form_errors.html" with errors=form.non_field_errors %}, and did this faithfully in every template where I couldn't just render {{ form }}. It took writing up this document to understand that I could just do {{ form.non_field_errors }} instead. This is on me: it’s mentioned in the Form topic documentation.
  • This might have been averted if either the form rendering API docs or the rendering part of the forms API docs referred back to the Reusable form guide more liberally. This is partly on me, but I think also partly on the docs – the reference is provided further up in the long section intro, but not on e.g. render() itself. If you’re familiar with the docs and their structure, you’ll look up render() and realise that this is a subsection and you should scroll up to the next h2 for details (h3 not being high up enough). But, alas, I just scrolled up to h3, and decided that the reference docs were just this sparse, not realising that was long-form text waiting for me.
  • Part of this is overwhelming docs: I did read the important part earlier when coming to terms with the whole template-based form rendering flow, but I didn’t know which parts were important then. When I wanted to refer back to things, I tended to become lost between the form rendering API docs, the form API docs, and the form topic docs (and I think at least one more page in the mix). Which is why I finally resorted to just working off the Django source code, because jumping between widget templates, form, field and BoundField definitions was more practical (but led to me overlooking important things like how error rendering works).
• While we’re at it: gripes with the form rendering API docs abound.
  • The docs are generally frustrating if you aren’t already looking at the code. Form.render() is noted to use the context from Form.get_context(), which is "form: the bound form, fields: all bound fields, except the hidden fields, hidden_fields: all hidden bound fields, errors: all non-field-related or hidden field related form errors.". Pop quiz, what’s in context["fields"]? If you said “a list of (field, error) tuples”, congratulations for understanding the obvious (??).
  • Another example: as introduction, the docs say "Use one of the built-in template form renderers or implement your own. Custom renderers must implement a render(template_name, context, request=None) method.". There’s no example for this, or any other guidance, or even a link to what the context is. IMO the least the docs could do here would be to point out that you can have a custom renderer without implementing render() yourself by building off TemplateRenderer – this is what’s recommended elsewhere, so maybe the docs assume you remember that this is the example given – or don’t give advice like this in reference documentation, I guess.
• The fact that default_renderer is set on Form instead on Form.Meta is so weird. Up to this point, things set directly on the form have been form fields, and things concerning presentation and other options went to Form.Meta. I’ll grant that the whole .Meta thing is weird, but it’s such a fundamental Django convention that suddenly having to mix fields and other stuff directly on the Form makes the code harder to read and more annoying to write. I’m not sure what’s up with this, but I dislike it, on a purely aesthetically opinionated basis.
• It’s still weird and unintuitive to me what’s set where:
  • {{ field.label_tag }} seems a nice presentation-level function to use instead of building the label yourself, but it uses the form's label_suffix attribute, which means the output ends with a : by default. However, I mostly encounter wanting to change the label suffix on a renderer basis: tabular forms don't need it, inline forms might. Setting yet another presentation attribute on the form, especially when it’s tightly coupled with the renderer, is annoying busywork, so my renderers all just get to render labels themselves.
  • Similarly, I mostly find myself wanting to change how/where placeholders are used on a per-renderer basis: Placeholders might make sense when we hide the field’s label, but not if we render the same form with a renderer that makes labels visible. But from the renderer, all I can do is to always render the placeholders and then hide them via CSS, as the placeholder is just part of the {{ field }} rendering process.
  • Similarly, I feel like it’s a renderer’s prerogative to not render every information that is available. A renderer that does not render help texts can make sense in a specific context. But Django hard-adds aria-describedby in BoundField of all places, so there’s … not really anything you can do about it being there. The “easiest” way to accommodate this is to have a base form class that checks if a renderer without help texts is used and then sets all the help texts to None. It was kind of frustrating to puzzle all this out (though that may be on me for not being patient enough with the docs, or not thinking the right way).
  • Similarly, and possibly showing off my confusion regarding the data vs presentation layer here; I’d sometimes prefer to pick a field’s widget on a renderer basis – a plain renderer that uses the most standard HTML inputs, vs an enhanced renderer that prioritises progressive enhancement or even JavaScript inputs (averting the pitchforks: this is often sensible in places where HTML5 inputs like type=range or type=color are badly standardised and not super accessible). What I really felt I wanted here was to change (in the scope of a renderer or just generally) the default widget for a given field without overriding templates in django/forms.
• Moving towards the Django-ish way of handling forms brought made me want to use {{ form.media }} again, but that's still not a good idea and very sad.

More things where I feel Django could do more to provide guidance on:

• You can override a form’s template_name – when do you do this, vs just looping over form fields / doing what you have to do in the template the form is used in? I’d guess that easily 90% of my forms are only used on one specific page. Is there an advantage to using template_name (keep form stuff scoped) or do the obvious disadvantages (more templates, less locality of behaviour) win? So do you want to do this for reused forms (like auth forms), for example?

Try iommi. It's radically less code and you can customize is more sanely.