Active colonization time estimates for the galaxy are invariably much shorter than the lifespan of the galaxy. Passive diffusion would take longer of course but empirically observed Oort cloud mixing intervals even this far from the core for the sun are on the order of 10^5 years. Lingam and Loeb (2018) calculate the delivery of amino acids between star systems but not timescales; still, they estimate an upper bound for the size of interstellar objects for our system (a 10 kilometer-radius asteroid) and Alpha Centauri (an Earth sized planet.) This is a second source pointing to the fact that we may be an interstellar backwater, uniquely un-exposed to evidence of replicators* relative to the stars around us.
Where intergalactic material transfer is concerned, of course given the distances involved we should expect the process to be slower, both in terms of at an absolute rate and moreso in terms of colonizing systems, since the ratio of number of incoming objects:number of systems to receive material will be quite low. That said, a) there are extragalactic stars in the Milky Way right now, and b) we're actually talking about an exponential rather than linear rate if there are replicators* of any sort being introduced. This excludes infrequent but massive events like intergalactic collisions, like those which the Milky Way has undergone repeatedly in the past.
*I deliberately use the term replicators as a catch-all to include "space-viroids" (most likely), von Neumann probes, "cancerous" (mutants selected for fecundity over original function) or otherwise, or deliberate colonizations by agents with some kind of intention (least likely.)