f(f(x)) = 4x

There are plenty of continuous functions such that . Besides the trivial examples and , one can take any equation that is symmetric in and has a unique solution for one variable in terms of the other. For example: leads to .

I can’t think of an explicit example that is also differentiable, but implicitly one can be defined by , for example. In principle, this can be made explicit by solving the cubic equation for , but I’d rather not.

At the time of writing, I could not think of any diffeomorphism such that both and have a nice explicit form. But Carl Feynman pointed out in a comment that the hyperbolic sine has the inverse which certainly qualifies as nice and explicit.

Let’s change the problem to . There are still two trivial, linear solutions: and . Any other? The new equation imposes stronger constraints on : for example, it implies

But here is a reasonably simple nonlinear continuous example: define

and extend to all by . The result looks like this, with the line drawn in red for comparison.

To check that this works, notice that maps to , which the function maps to , and of course .

From the plot, this function may appear to be differentiable for , but it is not. For example, at the left derivative is while the right derivative is .
This could be fixed by picking another building block instead of , but not worth the effort. After all, the property is inconsistent with differentiability at as long as is nonlinear.

The plots were made in Sage, with the function f define thus:

def f(x):
    if x == 0:
        return 0
    xa = abs(x)
    m = math.floor(math.log(xa, 2))
    if m % 2 == 0:
        return math.copysign(2**(m + xa/2**m), x)
    else:
        return math.copysign(2**(m+1) * (math.log(xa, 2)-m+1), x)