DragonX/ObsidianDragon
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity

perf(image-picker): decode thumbnails off the UI thread

Browse Source 
The image picker decoded up to 6 full-size images per frame on the UI
thread to make its thumbnails (1.5-2.7 s per frame with 12 MP photos,
until the visible grid filled) and read every .gif/.webp a second time
just to see whether it animates. Thumbnails were freed only on navigate
or close, so a large folder and every hovered animation (up to ~20 MB
each) stayed resident.

Thumbnails now decode on util::StartImageDecode workers (at most 2 at
once, so the top of the visible grid fills first) and the UI thread
uploads up to 8 small textures per frame. The animation badge comes from
the same decode, with no second read. Hover previews get their own
2-slot lane, and the picker asks for a frame every 100 ms while anything
is pending; under idle pacing a deferred thumbnail used to wait for input
or the 2 s tick.

Behaviour changes:
- thumbnails and hover frames are LRU-capped at 64 MiB per folder;
  evicted ones re-decode when they are needed again;
- leaving an animated thumbnail before its preview finishes decoding
  cancels it (re-hover restarts it); a finished preview is kept;
- animated WebP files get a real thumbnail instead of the placeholder;
- images above the decoder's limits show the placeholder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DanS
2026-10-09 21:09:48 -05:00
co-authored by Claude Opus 5.5
parent 3af7b0ef44
commit 4a2a1bdd54
1 changed files with 138 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ui/windows/image_picker.h
+138 -86
View File
@@ -7,8 +7,9 @@
// custom contact avatar. Like FolderPicker, only one overlay renders at a time (the framework does
// not nest), so the caller suppresses its own overlay while ImagePicker::isOpen().
//
// Thumbnails are decoded to raw pixels, box-downscaled to a small texture, and cached per directory
// (destroyed on navigate/close) so browsing a Pictures folder full of large photos stays cheap.
// Thumbnails are read, decoded and box-downscaled on background workers (util::StartImageDecode), then
// uploaded to small textures on the UI thread a few per frame, and cached per directory (destroyed on
// navigate/close, LRU-capped within a directory) so browsing a folder of large photos never stalls the UI.
#pragma once
@@ -29,6 +30,7 @@
#include "../../util/i18n.h"
#include "../../util/platform.h"
#include "../../util/image_decode_core.h"
#include "../../util/texture_loader.h"
#include "../../embedded/IconsMaterialDesign.h"
#include "../layout.h"
@@ -65,7 +67,7 @@ public:
using namespace material;
const float dp = Layout::dpiScale();
s_loadedThisFrame = 0; // budget: decode at most a few thumbnails per frame (no scroll hitch)
s_uploadsThisFrame = 0; // budget: upload at most a few finished thumbnails per frame (no scroll hitch)
ImFont* rowFont = Type().body1();
ImFont* icoFont = Type().iconSmall();
@@ -231,7 +233,7 @@ public:
ImGui::PopID();
dl->AddRectFilled(mn, mx, WithAlpha(OnSurface(), 18), 6.0f * dp);
// Only decode thumbnails for cells actually on-screen, and only a few per frame.
// Only cells actually on-screen start a (background) decode; finished ones upload a few per frame.
const Thumb* t = ImGui::IsRectVisible(mn, mx) ? thumbFor(abs) : nullptr;
ImTextureID tex = t ? t->tex : 0;
int tw = t ? t->w : 0, thh = t ? t->h : 0;
@@ -286,6 +288,7 @@ public:
}
}
maintainThumbs(); // after every Thumb* of this frame was used: cancel left hovers, LRU-evict
ImGui::EndChild(); // inner scrolling list
ImGui::PopStyleVar(2); // ScrollbarSize + ScrollbarRounding
ImGui::EndChild(); // outer frame
@@ -331,50 +334,44 @@ public:
static bool consumeAnimationActive() { bool v = s_animatingThisFrame; s_animatingThisFrame = false; return v; }
private:
// Background decode job: a detached worker fills `result` off the UI thread and sets `done`. The
// shared_ptr keeps it alive if the owning Thumb is destroyed mid-decode (no blocking on navigate).
struct AnimJob {
std::atomic<bool> done{false};
util::AnimFrames result;
};
// Decode jobs (util::StartImageDecode) run on detached workers that co-own the job: if the owning Thumb
// is erased mid-decode (navigate/close/evict) the worker stops early or finishes and frees it itself.
struct Thumb {
ImTextureID tex = 0; int w = 0, h = 0; bool tried = false; // static frame-0 thumbnail
bool animated = false; // multi-frame GIF/WebP (badge)
// Async animation load (kicked off on hover): the full frame decode runs on a worker thread,
// then frames are uploaded to textures a few per UI frame — so hovering never blocks the UI.
util::ImageDecodeJobPtr thumbJob; // still-thumbnail decode in flight
uint64_t lastUse = 0; // ImGui frame the cell was last drawn (LRU)
uint64_t hoverFrame = 0; // ImGui frame it was last hovered (LRU)
// Async animation load (kicked off on hover): the frame decode runs on a worker thread, then frames
// are uploaded to textures a few per UI frame — so hovering never blocks the UI.
bool animRequested = false, staged = false, animReady = false;
std::shared_ptr<AnimJob> job;
util::ImageDecodeJobPtr job; // hover-animation decode in flight
util::AnimFrames pending; // decoded frames awaiting GPU upload
size_t uploaded = 0;
std::vector<ImTextureID> frames;
std::vector<float> delays; int aw = 0, ah = 0; float totalDur = 0.0f;
};
// Drive the async animation load for a hovered thumbnail: (1) kick off a background decode once,
// (2) move its result to a staging buffer when done, (3) upload a few frames per UI frame. Never
// blocks — the static thumbnail keeps showing until the sequence is fully uploaded.
// Drive the async animation load for a hovered thumbnail: (1) start a background decode once (hover
// lane, at most 2 at once), (2) move its result to a staging buffer when done, (3) upload a few frames
// per UI frame. Never blocks — the static thumbnail keeps showing until the sequence is fully uploaded.
static void driveThumbAnim(Thumb& th, const std::string& path) {
// FIRST, before any early return: marks this thumb's frames as drawn this frame, so maintainThumbs()
// never destroys a playing animation in the frame that draws it.
th.hoverFrame = static_cast<uint64_t>(ImGui::GetFrameCount());
if (th.animReady) return;
if (!th.animRequested) {
if (s_animInFlight->load() >= kMaxAnimInFlight) return; // cap workers; retry a later frame
th.job = util::StartImageDecode(s_animLane, path, kThumbMaxFrames, util::ImageLimits{});
material::RequestProgressFrame(); // poll for the result (or a free slot) without input
if (!th.job) return; // both hover slots busy: retry next frame
th.animRequested = true;
th.job = std::make_shared<AnimJob>();
s_animInFlight->fetch_add(1);
std::shared_ptr<AnimJob> job = th.job;
std::shared_ptr<std::atomic<int>> inflight = s_animInFlight; // co-owned: safe past teardown
std::string p = path;
std::thread([job, inflight, p]() {
util::LoadAnimatedRGBA(p.c_str(), kThumbMaxFrames, job->result);
job->done.store(true, std::memory_order_release);
inflight->fetch_sub(1);
}).detach();
return;
}
if (th.job && th.job->done.load(std::memory_order_acquire)) {
th.pending = std::move(th.job->result);
if (th.job) {
if (!th.job->finished()) { material::RequestProgressFrame(); return; } // still decoding
if (th.job->ok()) th.pending = std::move(th.job->result);
th.job.reset();
if (th.pending.frames.size() <= 1) { // turned out not to animate
if (th.pending.frames.size() <= 1) { // turned out not to animate (or failed)
th.animated = false; th.pending = util::AnimFrames{}; th.animReady = true; return;
}
th.staged = true;
@@ -412,14 +409,6 @@ private:
return false;
}
// Only these can hold multiple frames in our decoders; skip the hover re-decode for other formats.
static bool isAnimatableName(const std::string& name) {
auto lo = name;
for (char& c : lo) c = (char)std::tolower((unsigned char)c);
auto ends = [&](const char* e){ std::string s(e); return lo.size()>s.size() && lo.compare(lo.size()-s.size(), s.size(), s)==0; };
return ends(".gif") || ends(".webp");
}
static std::string defaultStartDir() {
namespace fs = std::filesystem;
std::error_code ec;
@@ -429,56 +418,117 @@ private:
return home;
}
// Lazy, budgeted thumbnail loader: decodes to raw pixels, box-downscales to <=THUMB px, uploads a
// small texture. Returns null (placeholder icon shown) until the budget lets it load.
// Lazy thumbnail loader. A visible, uncached cell starts one background decode (file read, header probe
// + limits, decode, downscale to <= kThumbPx all run on a worker; at most 2 at once, so the top of the
// visible grid fills first) and shows the placeholder icon until it is done; the UI thread then uploads
// the small texture (<= kUploadsPerFrame per frame). Returns null while both decode slots are busy (the
// cell retries next frame). The worker also reports whether the source animates (badge), from the same
// bytes, so there is no second file read.
static const Thumb* thumbFor(const std::string& absPath) {
auto it = s_thumbs.find(absPath);
if (it != s_thumbs.end()) return &it->second;
if (s_loadedThisFrame >= kLoadsPerFrame) return nullptr; // defer to a later frame
s_loadedThisFrame++;
Thumb th; th.tried = true;
// Cheap animation probe (header/structure only) so animated items can show a badge without
// decoding all frames; only .gif/.webp can be animations in our decoders.
if (isAnimatableName(absPath)) th.animated = util::IsAnimatedImageFile(absPath.c_str());
int sw = 0, sh = 0;
unsigned char* raw = util::LoadRawPixelsFromFile(absPath.c_str(), &sw, &sh);
if (raw && sw > 0 && sh > 0) {
int dw = sw, dh = sh;
if (sw > kThumbPx || sh > kThumbPx) {
float s = (float)kThumbPx / (float)std::max(sw, sh);
dw = std::max(1, (int)(sw * s));
dh = std::max(1, (int)(sh * s));
}
std::vector<unsigned char> small((size_t)dw * dh * 4);
for (int y = 0; y < dh; y++) {
int sy0 = y * sh / dh, sy1 = std::max(sy0 + 1, (y + 1) * sh / dh);
for (int x = 0; x < dw; x++) {
int sx0 = x * sw / dw, sx1 = std::max(sx0 + 1, (x + 1) * sw / dw);
uint32_t r=0,g=0,b=0,a=0,n=0;
for (int yy = sy0; yy < sy1; yy++)
for (int xx = sx0; xx < sx1; xx++) {
const unsigned char* p = raw + ((size_t)yy * sw + xx) * 4;
r+=p[0]; g+=p[1]; b+=p[2]; a+=p[3]; ++n;
}
unsigned char* d = small.data() + ((size_t)y * dw + x) * 4;
d[0]=(unsigned char)(r/n); d[1]=(unsigned char)(g/n); d[2]=(unsigned char)(b/n); d[3]=(unsigned char)(a/n);
}
}
if (util::CreateRawTexture(small.data(), dw, dh, false, &th.tex)) { th.w = dw; th.h = dh; }
if (it == s_thumbs.end()) {
util::ImageLimits lim;
lim.stillMax = lim.animMax = kThumbPx;
util::ImageDecodeJobPtr job = util::StartImageDecode(s_thumbLane, absPath, /*maxFrames=*/1, lim);
if (!job) { material::RequestProgressFrame(); return nullptr; } // slots busy: placeholder, retry
Thumb th;
th.thumbJob = std::move(job);
it = s_thumbs.emplace(absPath, std::move(th)).first;
}
if (raw) util::FreeRawPixels(raw);
auto& slot = (s_thumbs[absPath] = th);
return &slot;
Thumb& th = it->second;
th.lastUse = static_cast<uint64_t>(ImGui::GetFrameCount());
if (th.thumbJob) {
if (!th.thumbJob->finished() || s_uploadsThisFrame >= kUploadsPerFrame) {
material::RequestProgressFrame(); // still decoding / upload budget spent: come back next frame
return &th;
}
const util::AnimFrames& r = th.thumbJob->result;
if (th.thumbJob->ok() && !r.frames.empty()) {
++s_uploadsThisFrame;
if (util::CreateRawTexture(r.frames[0].data(), r.w, r.h, false, &th.tex)) { th.w = r.w; th.h = r.h; }
th.animated = r.animated;
}
th.thumbJob.reset(); // frees the decoded pixels
th.tried = true; // failed / too large: placeholder, no retry (as before)
}
return &th;
}
static void releaseThumb(Thumb& th) {
if (th.thumbJob) th.thumbJob->cancel.store(true, std::memory_order_relaxed); // worker stops early
if (th.job) th.job->cancel.store(true, std::memory_order_relaxed);
if (th.tex) util::DestroyTexture(th.tex);
for (ImTextureID f : th.frames) if (f) util::DestroyTexture(f);
}
static void clearThumbs() {
for (auto& kv : s_thumbs) {
if (kv.second.tex) util::DestroyTexture(kv.second.tex);
for (ImTextureID f : kv.second.frames) if (f) util::DestroyTexture(f);
}
for (auto& kv : s_thumbs) releaseThumb(kv.second);
s_thumbs.clear();
}
// Free a thumbnail's hover-animation frames (GPU + staged CPU copies) but keep its still; the next
// hover decodes the animation again.
static void dropThumbAnim(Thumb& th) {
if (th.job) th.job->cancel.store(true, std::memory_order_relaxed);
th.job.reset();
for (ImTextureID f : th.frames) if (f) util::DestroyTexture(f);
th.frames.clear(); th.delays.clear();
th.pending = util::AnimFrames{};
th.uploaded = 0; th.aw = th.ah = 0; th.totalDur = 0.0f;
th.animRequested = th.staged = th.animReady = false;
}
static size_t stillBytes(const Thumb& th) { return th.tex ? (size_t)th.w * (size_t)th.h * 4u : 0u; }
static size_t animBytes(const Thumb& th) {
const size_t fw = (size_t)(th.aw > 0 ? th.aw : th.pending.w);
const size_t fh = (size_t)(th.ah > 0 ? th.ah : th.pending.h);
size_t bytes = (th.frames.size() + th.pending.frames.size()) * fw * fh * 4u; // GPU frames + staged CPU copies
if (th.job && th.job->finished()) { // a finished hover decode not consumed yet (the mouse left): evictable
const util::AnimFrames& r = th.job->result;
bytes += r.frames.size() * (size_t)r.w * (size_t)r.h * 4u;
}
return bytes;
}
// Per-frame upkeep, run after the grid has drawn (so every texture drawn this frame has lastUse/hoverFrame
// == this frame and is protected): (1) cancel a hover decode the mouse has left, freeing its slot for
// what the user is looking at now (a FINISHED one is kept for the next hover); (2) above
// kThumbBudgetBytes, evict least-recently-used hover-frame sets and still thumbnails (re-decoded if needed).
static void maintainThumbs() {
const uint64_t frame = static_cast<uint64_t>(ImGui::GetFrameCount());
size_t total = 0;
for (auto& kv : s_thumbs) {
Thumb& th = kv.second;
if (th.job && !th.job->finished() && th.hoverFrame + 1 < frame) {
th.job->cancel.store(true, std::memory_order_relaxed);
th.job.reset();
th.animRequested = false; // a later hover restarts it
}
total += stillBytes(th) + animBytes(th);
}
if (total <= kThumbBudgetBytes) return;
std::vector<util::CacheItem> items;
std::vector<const std::string*> keys;
items.reserve(s_thumbs.size() * 2);
keys.reserve(s_thumbs.size());
for (const auto& kv : s_thumbs) {
const size_t k = keys.size();
keys.push_back(&kv.first);
items.push_back({k * 2, stillBytes(kv.second), kv.second.lastUse}); // protected while visible
items.push_back({k * 2 + 1, animBytes(kv.second), kv.second.hoverFrame}); // protected while hovered
}
// Copy the victims' paths BEFORE mutating the map (erasing invalidates the key pointers).
std::vector<std::pair<std::string, bool>> victims; // (path, wholeEntry)
for (size_t key : util::selectEvictions(std::move(items), kThumbBudgetBytes, frame))
victims.emplace_back(*keys[key / 2], key % 2 == 0);
for (const auto& v : victims) {
auto it = s_thumbs.find(v.first);
if (it == s_thumbs.end()) continue;
if (v.second) { releaseThumb(it->second); s_thumbs.erase(it); }
else dropThumbAnim(it->second);
}
}
static void navigate(const std::string& dir) {
namespace fs = std::filesystem;
std::error_code ec;
@@ -507,18 +557,20 @@ private:
}
static constexpr int kThumbPx = 128; // max thumbnail dimension (downscaled)
static constexpr int kLoadsPerFrame = 6; // decode budget per frame (spreads a big folder over frames)
static constexpr int kUploadsPerFrame = 8; // finished thumbnails uploaded per frame (<= 64 KB each)
static constexpr int kThumbMaxFrames = 300; // cap frames per hovered animation
static constexpr int kMaxAnimInFlight = 2; // max concurrent background decodes
static constexpr size_t kThumbBudgetBytes = 64u * 1024u * 1024u; // resident thumbnail + hover-frame bytes
// Heap-owned so a detached worker (which decrements it) can safely outlive static teardown at
// process exit — the worker holds a shared_ptr copy, so it never touches a destroyed static.
static inline std::shared_ptr<std::atomic<int>> s_animInFlight = std::make_shared<std::atomic<int>>(0);
// Decode lanes: at most 2 thumbnail decodes and 2 hover-animation decodes at once (separate, so hover
// previews are never starved by a folder of thumbnails). A lane's counter is heap-owned and co-owned
// by its detached workers, so a worker still running at process exit never touches a destroyed static.
static inline util::ImageDecodeLane s_thumbLane{2};
static inline util::ImageDecodeLane s_animLane{2};
static inline bool s_open = false;
// Main-thread-only (set in render/driveThumbAnim, read+cleared in consumeAnimationActive); NOT atomic
// on purpose — do not read/write it from a worker thread.
static inline bool s_animatingThisFrame = false; // a hover-preview animation is playing
static inline int s_loadedThisFrame = 0;
static inline int s_uploadsThisFrame = 0;
static inline std::string s_dir;
static inline std::string s_selected;
static inline std::vector<std::string> s_subdirs;