I have looked for but did not find way to see if a tournament is worthwhile considering buyin vs payout looking at number rounds, advancers, etc. Any thoughts with formula?
Generally we start with the total prize money divided by the number of entrants. This gives you the EV of the tournament assuming everyone has an equal chance. If the EV is more than the entry fee, then we say the event has an overlay. If this EV is more than your total expenses for attending (entry fee, travel, food, lodging, etc.), then the event may be profitable for you. Adjusting the EV to account for your skill level vs the rest of the field is more of an art than a science, especially since, the first few times you play an event, you probably don't have enough information yet. However, if you assume that you are more skillful than the average participant, then you can probably safely conclude that your EV will be somewhat higher than the one calculated above.
Rebuys may or may not complicate things a bit. If the rebuys are simply empty seats in later sessions which are being sold then you can just count them as additional entries. If there are separate rebuy rounds and everyone is guaranteed a rebuy if they want one, or if everyone has the same chance of getting a rebuy, then you don't count them. Things only get more complicated from there. The general solution is to build the probability tree for each possible way you can make your way through the tournament and to add the outcomes for which you win money multiplied by their probabilities. As an exercise, this should give you the same answer as the simplifications above in the cases where they apply.
There definitely are a few previous threads on this subject, but I too am finding it difficult to locate them.
A short cut for the tournament with re-buys is to include part of it in the entry fee. For example, if you know from the casino how many re-buys are generally sold, then you can estimate the average re-buy for a given player. Then the cost of the tournament will be entry fee+average re-buy+other costs of travel etc.