Mdthbs

TeXLive 2019 is here :D

One of its awaited features is the new expanded primitive now available in all of the main engines (only available in LuaTeX in previous years) thanks to Joseph Wright.

What we had up to now was:

• Full expansion with edef and the like;
  
  Pros: expands the entire token list respecting protected (any form thereof) tokens
  
  Cons: being an assignment operation, it is not itself expandable;




• Partial expansion exploiting romannumeral, which expands tokens while looking for an ⟨integer⟩ and a trailing ⟨optional space⟩;
  
  Pros: is expandable, allowing one¹ to write some really interesting macros (including an expandable emulation of the expanded primitive);
  
  Cons: only expands the “head” of the token list, stopping as soon as it finds an expandable token which does not fit into its argument specification;

expanded gives a nice mix of both, allowing us to get the full expansion of a token list while being itself expandable, apparently covering a great deal of the use-cases of both edef and romannumeral.

Of course edef still is really useful when the expansion of the tokens needs to be stored in a macro, and the produced macro can take arguments normally, which aren't features of expanded.

The romannumeral trick, however, seems to have lost much of it's usefulness with expanded. From what I understand, many situations which required romannumeral to get the full expansion of something, now can use expanded for the same effect with a proper expansion primitive.

I know that the answer to the question in the title is “no” because there are a few situations in which romannumeral is still useful. The main one, I think, are undelimited arguments; expanded requires the material to be expanded as a ⟨general text⟩ argument, while romannumeral can be put in the token and act indefinitely, until an unexpandable token is found. Another situation is, of course, when one really wants only the head of the token list expanded, then romannumeral is the tool for the job.

But both these situations seem much smaller than the entire scope of using romannumeral for expansion. Thus my question (finally) is: can the expansion trick with romannumeral be replaced mostly everywhere (as an example I can mention the LaTeX3 kernel, which heavily uses this romannumeral-powered expansion)? What other situations require romannumeral for expansion and will not work with expanded?

¹In this case a very particular one :)

In many case the romannumeral trick is obsolete, but there are at least two cases where it still is useful.

The first is in that as mentioned in the question it's 'open ended', whereas expanded requires an argument to be grabbed. The second is that romannumeral expansion works through tokens exactly as TeX itself does when typesetting, up to the point it finds some 'payload'. This allows us to deal with any expandable material and then 'look ahead' at the first non-expandable token. An example from l3galley:

cs_new_protected:Npn galley_par:

  {

    s__galley_par_omit

    exp_after:wN __galley_par_auxi: exp:w exp_end_continue_f:w

  }

cs_new_protected:Npn __galley_par_auxi:

  {

    peek_meaning:NTF s__galley_par_omit

      { __galley_par_aux:N }

      { __galley_par_auxii: }

  }

cs_new_protected:Npn __galley_par_aux:N #1

  {

    str_if_eq:nnF {#1} { s__galley_par_omit }

      {

        __galley_par_auxii:

        #1

      }

  }

The romannumeral here is hidden inside exp:w, but the idea is simple. The expansion will continue until some non-expandable token is found. We then 'come back' to a look-ahead, which checks for a token that is exactly equivalent to s__galley_par_omit, a marker for consecutive par tokens. This needs the open-ended nature of romannumeral and the fact that it stops.

It's also worth noting that romannumeral ignores protected status: that shows up above, and would also be handy in implementing for example halign in macros. (This is another place we need a look-ahead.)

These aspects are quite specialised, and in most cases do not apply. The bigger barrier to shifting to expanded is that it is only now generally available: it will be many years before it is safe to assume that expanded is safe to use without a fall-back.