fell into a lesswrong rabbit hole after re-reading paul christiano's 'what failure looks like' (it came up at the curve and i am struggling to understand 'loss of control' scenarios) and man all the decision theory stuff just instantly runs into the halting problem right away

determining whether a given program will output a particular symbol for particular inputs is semi-decidable and therefore equivalent to the halting problem. any 'read the other agent's source code'-based decision theory falls apart because of this fact.

in fact any *reflective* decision theory falls apart because of this fact. tdt et. al. only work in situations where agents are not turing complete. not very useful.

the logical uncertainty stuff holds up pretty well though

@azsantosk actually here’s a wild and unsubstantiated take; adding probability takes us from a discrete to a continuous setting and that’s what does away with undecidability that would be such a beautiful theorem

@azsantosk so either we are above or below turing

@azsantosk either omega has hypercomputation (is a wizard) or our agents aren’t very interesting

@azsantosk certainly then i am not capable of self-modification and many other things presupposed by the various decision theories on less wrong dot com