<p>I just learned quite a bit from Buffy's wonderful answer! I would only add that a <em>token</em> is simple the smallest unit of text that carries semantic meaning to the compiler.  Thus,</p>

<p><code>int func(int input){</code> has 7 tokens (four word tokens and three symbol tokens). An escape character in a string such as <code>\"</code>, then, is a note to the tokenizer.</p>

<p>How they are later grouped and interpreted are decisions for later in the compilation process, based on the contents of each token.</p>

<p>I don't think that I would try to relate token to either of your other terms, as the relationship is in no way straightforward.  Expressions can be tokens (such as <code>42</code>), or can be groups of tokens (<code>x + 5</code>). That depends on the contents of the tokens themselves.</p>

CC BY-SA 4.0
2019-10-07T20:50:11.680Z
Ah yes, tokens are syntactic elements (the scanner handles these). The other three are semantic elements. The parser handles them structurally and the semantic analyzer imputes meaning to them.
Buffy
CC BY-SA 4.0
2019-10-08T12:16:06.360Z
I know expressions, statements & blocks end up first decomposed to tokens then after subsequent analysis meaning is imputted to. But that's not the question. My students are asking if tokens are the small elements from which these more "sophisticated" combinations are built, shouldn't they conceptually be grouped in a category of their own even if for learning purposes? Hence, as social scientists used to arguing over concepts, they demanded a category to embrace these "higher" programming constructs. Do we have one or can someone attempt to coin one for me?
Mallam Awal
CC BY-SA 4.0
2019-10-10T06:55:30.490Z
Characters & strings of characters (used in tokenizing) are smaller syntactic & semantic units than tokens. You yourself have just talked about characters that mean something to the compiler. We normally talk about a "pass" of a compiler for which we posit new atomic parts, non-atomic groupings, input & output.
philipxy
CC BY-SA 4.0
2019-10-11T12:27:21.947Z
Yea, philipxy, the atomic/non-atomic characterization is    a close match. However, non-atomic applies to a much larger set than expressions, statements  & blocks. That's why I hesitated to use that. However, as you said earlier  catgories are. made as needed even if imperfect. Perhaps I'll settle for that for now then keep searching and thinking of a tighter term. Thanks
Mallam Awal
CC BY-SA 4.0
2019-10-07T20:46:48.117Z
2019-10-07T20:50:55.020Z
2019-10-07T20:50:55.020Z
Ben I.
<p>People create categories as needed to manage complexity. A language designer has defined syntactic &amp; semantic things--including categories of things--relevant to their purposes &amp; so can anyone--specific or fuzzy. It's unhelpful &amp; misconceived to expect that somehow categories are out there to be uncovered or that everything must be in a category or that any pattern one gets a glimpse of is evidence of a category out there to be uncovered. This is a case of the fallacy of Reification. Of mathematics, Kronecker said, "God made the integers; all else is the work of man."</p>

<p>In programming languages, expressions, statements and blocks are categories of language strings or trees with associated grammar production rules. Relevant helpful larger grouping for these are things like definitions, programs &amp; compilation units. But those are ad hoc per the language.</p>

<p>Generally, what helpful syntactic categories there are is driven by what helpful semantic categories there are. Sadly, programming languages' semantics are typically poorly taught. Even though its semantics are the raison d'etre of a language.</p>

CC BY-SA 4.0
2019-10-10T06:45:02.453Z
2019-10-10T06:51:08.153Z
2019-10-10T06:51:08.153Z
philipxy
<p>This is a surprisingly complex question. In many languages one can distinguish between <em>commands</em> and <em>expressions</em>. A command is something that changes the state of the computation. An expression is something that returns a value, but doesn't change the state. This is actually a good mental model even in languages that can confuse the two ideas in small or large ways. </p>

<p>Some languages consider statements to be commands, but in C, for example, an assignment statement is also an expression that has a value equal to the value assigned to the variable on the left. That may or may not be the value of the expression on the right, actually, as when a short is assigned to a long. </p>

<p>Likewise blocks can either be commands or expressions in most languages. A function block is an expression, returning what the function returns (more or less) but it can also have command-like properties if it changes the state generally, say by modifying a non-local variable. But the if-clause in an if statement is usually just a command. </p>

<p>So, I don't think there is a single unifying category for all three of your items, but the distinction between <em>commands</em> and <em>expressions</em> is useful. </p>

<p>It is useful for another reason as well. In C a "function" can return a value or not (void functions). But programs are easier to understand if a function that returns a value does <em>not</em> also change anything in the non-local state. Then functions when invoked will be either expressions (the value returned) or commands (the state changes) but never both. </p>

<p>But C also has a distinction that even command-like functions often return an error state indicator, zero if there are no errors. So common practice gets in the way of the useful rule in the paragraph above. C++ can use exceptions instead of error codes, of course. </p>

<p>But in more "modern" languages, the distinction between expressions and commands is maintained with a bit more clarity. Java, for example. </p>

CC BY-SA 4.0
2019-10-07T20:36:26.520Z
2019-10-07T20:36:26.520Z
Buffy
<p>I was teaching an intro to Programming class, full of social scientists. After explaining that the basic building blocks of a program are called tokens, I took them through Keywords, Identifiers, Literals etc. </p>

<p>I was asked a question. And have been reading to find answers in books and online. 
They asked if we group the basic elements of a program under the word "token", then what word groups the other program elements. e.g. expressions, statements and blocks, that are derived from tokens?</p>

<p>Help with a reference or a term that classifies these constructs as a group.</p>

CC BY-SA 4.0
2019-10-11T19:16:07.447Z
I wonder whether I just managed now to parse your question as you intended. Are you asking, what is the name for non-token language element? In a grammar the literals (eg tokens) are "terminals" & everything else (rule or its associated set of expressions/strings/trees) is a "non-terminal". But I had thought from your last line you that were asking about acategory containing just expression/statement/block. (Whatever you think of those as.) Can you clarify?
philipxy
CC BY-SA 4.0
2019-10-18T11:52:54.693Z
Well, what about "program elements" ?
Michel Billaud
CC BY-SA 4.0
2019-10-26T01:14:43.230Z
Program elements is the main topic. The issue is some program elements are called tokens and these are the atomic pieces, what will those pieces aggregated into expressions, statements and blocks be designated by collectively?
Mallam Awal
CC BY-SA 4.0
2019-10-27T02:16:15.923Z
Perhaps "Token groups" covers it. Or, perhaps "Compound tokens" would fit. My preference is for "Expressions", "Statements", "Blocks". Each is different, yet composed of tokens. In social science speak, you have "taboos", "mores" and "norms". Similar functions, culturally, without being in the same "classification" without creating one artificially.
Gypsy Spellweaver
CC BY-SA 4.0
2019-10-29T00:20:15.627Z
I like the compound tokens term. Comes closest to what I'm looking for.
Mallam Awal
<programming><syntax>
A category to classify expressions, statements and blocks
# A category to classify expressions, statements and blocks

(Asked by: username_0 on 2019-10-07)

I was teaching an intro to Programming class, full of social scientists. After explaining that the basic building blocks of a program are called tokens, I took them through Keywords, Identifiers, Literals etc.

I was asked a question. And have been reading to find answers in books and online.
They asked if we group the basic elements of a program under the word "token", then what word groups the other program elements. e.g. expressions, statements and blocks, that are derived from tokens?

Help with a reference or a term that classifies these constructs as a group.

**Tags:** programming, syntax

> **Comments:**
> **username_1**: I wonder whether I just managed now to parse your question as you intended. Are you asking, what is the name for non-token language element? In a grammar the literals (eg tokens) are "terminals" & everything else (rule or its associated set of expressions/strings/trees) is a "non-terminal". But I had thought from your last line you that were asking about acategory containing just expression/statement/block. (Whatever you think of those as.) Can you clarify?  
> **username_2**: Well, what about "program elements" ?  
> **username_0**: Program elements is the main topic. The issue is some program elements are called tokens and these are the atomic pieces, what will those pieces aggregated into expressions, statements and blocks be designated by collectively?  
> **username_3**: Perhaps "Token groups" covers it. Or, perhaps "Compound tokens" would fit. My preference is for "Expressions", "Statements", "Blocks". Each is different, yet composed of tokens. In social science speak, you have "taboos", "mores" and "norms". Similar functions, culturally, without being in the same "classification" without creating one artificially.  
> **username_0**: I like the compound tokens term. Comes closest to what I'm looking for.  

---

## 3 answers

### Answer by username_1 on 2019-10-10. Score: 1 | Accepted answer

People create categories as needed to manage complexity. A language designer has defined syntactic & semantic things--including categories of things--relevant to their purposes & so can anyone--specific or fuzzy. It's unhelpful & misconceived to expect that somehow categories are out there to be uncovered or that everything must be in a category or that any pattern one gets a glimpse of is evidence of a category out there to be uncovered. This is a case of the fallacy of Reification. Of mathematics, Kronecker said, "God made the integers; all else is the work of man."

In programming languages, expressions, statements and blocks are categories of language strings or trees with associated grammar production rules. Relevant helpful larger grouping for these are things like definitions, programs & compilation units. But those are ad hoc per the language.

Generally, what helpful syntactic categories there are is driven by what helpful semantic categories there are. Sadly, programming languages' semantics are typically poorly taught. Even though its semantics are the raison d'etre of a language.


---

### Answer by username_5 on 2019-10-07. Score: 2

This is a surprisingly complex question. In many languages one can distinguish between *commands* and *expressions*. A command is something that changes the state of the computation. An expression is something that returns a value, but doesn't change the state. This is actually a good mental model even in languages that can confuse the two ideas in small or large ways.

Some languages consider statements to be commands, but in C, for example, an assignment statement is also an expression that has a value equal to the value assigned to the variable on the left. That may or may not be the value of the expression on the right, actually, as when a short is assigned to a long.

Likewise blocks can either be commands or expressions in most languages. A function block is an expression, returning what the function returns (more or less) but it can also have command-like properties if it changes the state generally, say by modifying a non-local variable. But the if-clause in an if statement is usually just a command.

So, I don't think there is a single unifying category for all three of your items, but the distinction between *commands* and *expressions* is useful.

It is useful for another reason as well. In C a "function" can return a value or not (void functions). But programs are easier to understand if a function that returns a value does *not* also change anything in the non-local state. Then functions when invoked will be either expressions (the value returned) or commands (the state changes) but never both.

But C also has a distinction that even command-like functions often return an error state indicator, zero if there are no errors. So common practice gets in the way of the useful rule in the paragraph above. C++ can use exceptions instead of error codes, of course.

But in more "modern" languages, the distinction between expressions and commands is maintained with a bit more clarity. Java, for example.


---

### Answer by username_4 on 2019-10-07. Score: 0

I just learned quite a bit from username_5's wonderful answer! I would only add that a *token* is simple the smallest unit of text that carries semantic meaning to the compiler. Thus,

`int func(int input){` has 7 tokens (four word tokens and three symbol tokens). An escape character in a string such as `\"`, then, is a note to the tokenizer.

How they are later grouped and interpreted are decisions for later in the compilation process, based on the contents of each token.

I don't think that I would try to relate token to either of your other terms, as the relationship is in no way straightforward. Expressions can be tokens (such as `42`), or can be groups of tokens (`x + 5`). That depends on the contents of the tokens themselves.

> **Comments:**
> **username_5**: Ah yes, tokens are syntactic elements (the scanner handles these). The other three are semantic elements. The parser handles them structurally and the semantic analyzer imputes meaning to them.  
> **username_0**: I know expressions, statements & blocks end up first decomposed to tokens then after subsequent analysis meaning is imputted to. But that's not the question. My students are asking if tokens are the small elements from which these more "sophisticated" combinations are built, shouldn't they conceptually be grouped in a category of their own even if for learning purposes? Hence, as social scientists used to arguing over concepts, they demanded a category to embrace these "higher" programming constructs. Do we have one or can someone attempt to coin one for me?  
> **username_1**: Characters & strings of characters (used in tokenizing) are smaller syntactic & semantic units than tokens. You yourself have just talked about characters that mean something to the compiler. We normally talk about a "pass" of a compiler for which we posit new atomic parts, non-atomic groupings, input & output.  
> **username_0**: Yea, username_1, the atomic/non-atomic characterization is    a close match. However, non-atomic applies to a much larger set than expressions, statements  & blocks. That's why I hesitated to use that. However, as you said earlier  catgories are. made as needed even if imperfect. Perhaps I'll settle for that for now then keep searching and thinking of a tighter term. Thanks  

---

