Summary

You have built from transistors what a processor counts with.

The half adder — XOR and AND: the sum and the carry of one digit.
The full adder takes a carry from its neighbour; four and eight of them are column addition, the carry runs through the digits.
Two's complement — negative numbers with no new circuit: −B is the flipped bits plus 1. Subtraction is the same adder with XOR at its input.
Flags C, Z, N, V — what the processor knows about the answer. Comparison CMP is a subtraction that leaves only the flags.
The ± calculator — one more subtraction, 0 − S, turns the answer into a sign and a magnitude.
Decimal math (bonus) — the "+6" correction after every digit.

The finished calculator is 1,208 transistors.

Fast carry. In our adder the carry runs through the digits one by one: the top digit waits for all the lower ones. For 8 bits you do not notice it, for 64 it is slow. Real processors work out in advance whether a carry will be born in a group of digits, and do not wait for the wave.
Multiplication is addition with shifts: 13 × 5 = 13 × 4 + 13 × 1. An adder, a register and a shift — and the machine multiplies in columns, like at school.
The ALU. Add AND, OR, XOR and a choice of result to the adder — and you get the arithmetic logic unit, the heart of a processor.